首页 > 其他分享 >GreenPlum安装

GreenPlum安装

时间：2023-04-03 11:14:00浏览次数：35  
标签：aivmp 安装 GreenPlum greenplum master home data gpadmin

GreenPlum安装指南

针对Centos7平台

[gpadmin@aivmp-bigdata0001 ~]$ cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

机器

  • 外网地址
121.36.194.125 aivmp-bigdata0001
124.70.139.178 aivmp-bigdata0002
124.70.192.187 aivmp-bigdata0003
121.36.245.248 ecs-application0002
root/Allcam@2021
  • 内网地址
192.168.0.244   aivmp-bigdata0001
192.168.0.81    aivmp-bigdata0002
192.168.0.128   aivmp-bigdata0003
192.168.0.150 ecs-application0002

禁用SELinux和防火墙软件

对于运行RHEL或CentOS的所有Greenplum数据库主机系统，必须禁用SELinux。跟着这些步骤：

  1. 以root用户身份检查SELinux的状态：

    [gpadmin@aivmp-bigdata0001 ~]# sestatus
SELinux status:                 disabled

  2. 如果未禁用SELinux，请通过编辑/etc/selinux/config文件。以root身份更改文件如下：

    SELINUX=disabled

  3. 如果系统上安装了系统安全服务守护程序（SSSD），请编辑SSSD配置文件并设置

    selinux_provider 的参数 .以防止即使禁用SELinux也可能发生与SELinux相关的SSH身份验证拒绝。以root用户身份，编辑/etc/sssd/sssd.conf 并添加以下参数：

    selinux_provider=none

  4. 重新引导系统以应用所做的任何更改，并验证SELinux已禁用。

您还应该禁用防火墙软件,请按照以下步骤禁用 iptables

  1. 对于具有防火墙，检查状态防火墙。使用命令：

    [gpadmin@aivmp-bigdata0001 ~]$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

  2. 如有必要，以root用户身份执行这些命令以禁用防火墙：

    systemctl stop firewalld.service
systemctl disable firewalld.service

配置Hosts

# cat >> /etc/hosts <<EOF
192.168.0.244 aivmp-bigdata0001
192.168.0.81 aivmp-bigdata0002
192.168.0.128 aivmp-bigdata0003
192.168.0.150 ecs-application0002
EOF

ping 192.168.0.244
ping 192.168.0.81 
ping 192.168.0.128
ping 192.168.0.150

修改内核文件参数

vi /etc/sysctl.conf

cat >> /etc/sysctl.conf << EOF
#xialonglong
kernel.shmall = 4097404
kernel.shmmax = 16782966784
kernel.shmmni = 4096
vm.overcommit_memory = 2
vm.overcommit_ratio = 95

net.ipv4.ip_local_port_range = 10000 65535
kernel.sem = 500 2048000 200 4096
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.ip_local_reserved_ports=65330 
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.all.arp_filter = 1
net.core.netdev_max_backlog = 10000
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
vm.min_free_kbytes = 983377
vm.swappiness = 10
vm.zone_reclaim_mode = 0
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100
vm.dirty_background_ratio = 3
vm.dirty_ratio = 10
EOF

echo 1024 > /proc/sys/net/core/somaxconn
echo "4096 65536 16777216"> /proc/sys/net/ipv4/tcp_wmem
echo 360000 > /proc/sys/net/ipv4/tcp_max_tw_buckets

修改系统连接数

vi /etc/security/limits.conf

# cat > /etc/security/limits.conf<< EOF
* soft nofile 524288
* hard nofile 524288
* soft nproc 131072
* hard nproc 131072
EOF

cat > /etc/security/limits.conf<< EOF
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
EOF

cat /etc/security/limits.conf

ulimit -u
sysctl -p

ulimit -n

//设置read-ahead

/sbin/blockdev --setra 16384 /dev/vdb1
/sbin/blockdev --setra 16384 /dev/vda1

//设置IO调度策略

[gpadmin@aivmp-bigdata0001 ~]$ cat /sys/block/vda/queue/scheduler
[mq-deadline] kyber none

#grubby --update-kernel=ALL --args="elevator=deadline"

//禁用Transparent Huge Page(THP)

#grubby --update-kernel=ALL --args="transparent_hugepage=never"

//设置RemoveIPC

[gpadmin@aivmp-bigdata0001 ~]$ cat /etc/systemd/logind.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See logind.conf(5) for details.

[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
#HandleLidSwitch=suspend
#HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
RemoveIPC=no
#UserTasksMax=

//增加ssh最大连接数

/etc/ssh/sshd_config

# cat >> /etc/ssh/sshd_config<< EOF
MaxStartups 200
MaxSessions 200
EOF
# service sshd restart

同步集群时间

yum install ntp ntpdate -y
sudo ntpdate cn.pool.ntp.org && hwclock --systohc

创建Greenplum管理用户

  1. 创建 gpadmin组和用户。

    groupadd gpadmin 
useradd gpadmin -r -m -g gpadmin 
passwd gpadmin
新密码：<gpadmin>
重新输入新密码：<gpadmin>

  2. 切换到 gpadmin用户并对 gpadmin 用户为该服务器生成SSH密钥。

    $ su gpadmin 
$ ssh-keygen -t rsa
$ ssh-copy-id aivmp-bigdata0001
$ ssh-copy-id aivmp-bigdata0002
$ ssh-copy-id aivmp-bigdata0003
$ ssh-copy-id ecs-application0002

  3. （可选）对 gpadmin 用户授予sudo权限。

    # visudo 
.....
root    ALL=(ALL)       ALL
gpadmin ALL=(ALL)       ALL
.....

  4. 添加 gpadmin 用户到此分组。

    usermod -aG wheel gpadmin

安装数据库到每台机器

  1. 下载安装
wget https://github.com/greenplum-db/gpdb/releases/download/6.14.1/open-source-greenplum-db-6.14.1-rhel7-x86_64.rpm && yum install open-source-greenplum-db-6.14.1-rhel7-x86_64.rpm -y
  1. 授予权限给gpadmin用户
sudo chown -R gpadmin:gpadmin /usr/local/greenplum*
sudo chgrp -R gpadmin /usr/local/greenplum*
  1. 添加环境变量为gpadmin
$ cat > /home/gpadmin/.bashrc<< EOF
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/data/master/gpseg-1
export PGPORT=5432
export PGUSER=gpadmin
export PGDATABASE=test
export LD_PRELOAD=/lib64/libz.so.1 ps
export GPHOME=/usr/local/greenplum-db
EOF


cat >> /home/gpadmin/.bashrc<< EOF
source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/data/master/gpseg-1
export PGPORT=5432
export PGUSER=gpadmin
export PGDATABASE=test
export LD_PRELOAD=/lib64/libz.so.1 ps
export GPHOME=/usr/local/greenplum-db
EOF

cat ~/.bashrc

master机器

  1. 创建文件allhost
$ cat > /usr/local/greenplum-db/allhost<< EOF
aivmp-bigdata0001
aivmp-bigdata0002
aivmp-bigdata0003
ecs-application0002
EOF
  1. 创建文件seghost
$ cat > /usr/local/greenplum-db/seghost<< EOF
aivmp-bigdata0001
aivmp-bigdata0002
aivmp-bigdata0003
EOF

cat > /usr/local/greenplum-db/seghost<< EOF
aivmp-bigdata0002
aivmp-bigdata0003
EOF


cat > /usr/local/greenplum-db/seghost<< EOF
node203
node204
EOF

cat > /usr/local/greenplum-db/seghost<< EOF
cdh-upgrade-1
cdh-upgrade-3
EOF

cat > /usr/local/greenplum-db/seghost<< EOF
cdh-upgrade-3
EOF
  1. 创建文件夹
# mkdir -p /data/master
# chown -R gpadmin:gpadmin /data/master
  1. 创建gpinitsystem_config文件

注意：declare -a DATA_DIRECTORY和declare -a MIRROR_DATA_DIRECTORY必须是成对出现得。里面几个文件夹，就代表一个segment机器上面启动了几个实例。一般推荐每台机器上启动2~4个实例，最大化cpu利用。

cat > /home/gpadmin/gpinitsystem_config<< EOF
ARRAY_NAME="Greenplum Data Platform"
SEG_PREFIX=gpseg
PORT_BASE=6000
declare -a DATA_DIRECTORY=(/home/gpadmin/data/primary /home/gpadmin/data/primary)
MASTER_HOSTNAME=ecs-application0002
MASTER_DIRECTORY=/home/gpadmin/data/master
MASTER_PORT=5432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=7000
declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/data/mirror /home/gpadmin/data/mirror)
DATABASE_NAME=test
MACHINE_LIST_FILE=/usr/local/greenplum-db/seghost
EOF



cat > /home/gpadmin/gpinitsystem_config<< EOF
ARRAY_NAME="Greenplum Data Platform"
SEG_PREFIX=gpseg
PORT_BASE=6000
declare -a DATA_DIRECTORY=(/home/gpadmin/data/primary /home/gpadmin/data/primary /home/gpadmin/data/primary)
MASTER_HOSTNAME=ecs-application0002
MASTER_DIRECTORY=/home/gpadmin/data/master
MASTER_PORT=5432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=7000
declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/data/mirror /home/gpadmin/data/mirror /home/gpadmin/data/mirror)
DATABASE_NAME=test
MACHINE_LIST_FILE=/usr/local/greenplum-db/seghost
EOF

cat > /home/gpadmin/gpinitsystem_config<< EOF
ARRAY_NAME="Greenplum Data Platform"
SEG_PREFIX=gpseg
PORT_BASE=6000
declare -a DATA_DIRECTORY=(/data/primary /data/primary)
MASTER_HOSTNAME=ecs-application0002
MASTER_DIRECTORY=/data/master
MASTER_PORT=5432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=7000
declare -a MIRROR_DATA_DIRECTORY=(/data/mirror /data/mirror)
DATABASE_NAME=test
MACHINE_LIST_FILE=/usr/local/greenplum-db/seghost
EOF


cat > /home/gpadmin/gpinitsystem_config<< EOF
ARRAY_NAME="Greenplum Data Platform"
SEG_PREFIX=gpseg
PORT_BASE=8000
declare -a DATA_DIRECTORY=(/data/primary /data/primary)
MASTER_HOSTNAME=node201
MASTER_DIRECTORY=/data/master
MASTER_PORT=15432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=9000
declare -a MIRROR_DATA_DIRECTORY=(/data/mirror /data/mirror)
DATABASE_NAME=test
MACHINE_LIST_FILE=/usr/local/greenplum-db/seghost
EOF


cat > /home/gpadmin/gpinitsystem_config<< EOF
ARRAY_NAME="Greenplum Data Platform"
SEG_PREFIX=gpseg
PORT_BASE=16000
declare -a DATA_DIRECTORY=(/data/primary /data/primary)
MASTER_HOSTNAME=cdh-upgrade-2
MASTER_DIRECTORY=/data/master
MASTER_PORT=15432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=17000
declare -a MIRROR_DATA_DIRECTORY=(/data/mirror /data/mirror)
DATABASE_NAME=test
MACHINE_LIST_FILE=/usr/local/greenplum-db/seghost
EOF

cat > /home/gpadmin/gpinitsystem_config<< EOF
ARRAY_NAME="Greenplum Data Platform"
SEG_PREFIX=gpseg
PORT_BASE=16000
declare -a DATA_DIRECTORY=(/data/primary /data/primary)
MASTER_HOSTNAME=cdh-upgrade-3
MASTER_DIRECTORY=/data/master
MASTER_PORT=15432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=17000
declare -a MIRROR_DATA_DIRECTORY=(/data/mirror /data/mirror)
DATABASE_NAME=test
MACHINE_LIST_FILE=/usr/local/greenplum-db/seghost
EOF

segment机器

# source /usr/local/greenplum-db/greenplum_path.sh 
# gpssh -f seghost -e 'mkdir -p /data/primary'
# gpssh -f seghost -e 'mkdir -p /data/mirror'
# gpssh -f seghost -e 'chown -R gpadmin:gpadmin /data/*'
$ gpssh-exkeys -f /usr/local/greenplum-db/seghost
$ gpssh-exkeys -f /usr/local/greenplum-db/allhost

sudo chown -R gpadmin:gpadmin /data

mkdir -p /data/primary /data/mirror /data/master
mkdir -p data/mirror
mkdir -p data/master

初始化

  1. 正常的初始化
$ gpinitsystem -c gpinitsystem_config
  1. 带有Standby节点：
  • Standby节点必须能ssh到所有机器，其他配置和master保持完全一致，同样需要提前建立data/master/文件夹。
$ gpinitsystem -c  gpinitsystem_config -s aivmp-bigdata0001
  • 如果初始化得时候没有做Standby，那就在集群正常运行得时候，在master节点运行gpinitstandby命令完成新增Stanby节点
$ gpinitstandby -s aivmp-bigdata0001
  • 不成功了删除Stanby产生得脏数据，排错，从新建立Standby节点
$ rm -rf /home/gpadmin/data/master/*
  1. 初始化失败后，清楚数据，清楚日志，并重新初始化
# rm -rf  /home/gpadmin/data/master/* /home/gpadmin/data/primary/* /home/gpadmin/data/mirror/* /home/gpadmin/gpAdminLogs /tmp/*

# rm -rf  /data/master/* /data/primary/* /data/mirror/* /home/gpadmin/gpAdminLogs /tmp/*

# rm -rf  /home/gpadmin/data /home/gpadmin/gpAdminLogs /tmp/*

授权远程连接

cat >> /data/master/gpseg-1/pg_hba.conf<<EOF
host     all         gpadmin         221.226.11.218/32       trust
host     replication gpadmin         221.226.11.218/32       trust
EOF

cat >> /data/master/gpseg-1/pg_hba.conf<<EOF
host    all            all           0.0.0.0/0               trust
EOF

pg_ctl reload -D /data/master/gpseg-1/

cat >> /home/gpadmin/data/master/gpseg-1/pg_hba.conf<<EOF
host     all         gpadmin         221.226.11.218/32       trust
host     replication gpadmin         221.226.11.218/32       trust
EOF


cat >> /home/gpadmin/data/master/gpseg-1/pg_hba.conf<<EOF
host     all         gpadmin         221.226.11.218/32       trust
host     replication gpadmin         221.226.11.218/32       trust
host    all            all           0.0.0.0/0               trust
EOF

cat >> /home/gpadmin/data/master/gpseg-1/pg_hba.conf<<EOF
host    all            all           0.0.0.0/0               trust
EOF

pg_ctl reload -D /home/gpadmin/data/master/gpseg-1/

报错

  1. 如果初始化stanby节点报错：

-[ERROR]:-Failed to copy data directory from master to standby.

20200331:15:20:51:008596 gpinitstandby:pc3:gpadmin-[ERROR]:-Failed to create standby

[ERROR]:-Error initializing standby master: ExecutionError: 'non-zero rc: 1' occurred. Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 pc1 ". /usr/local/greenplum-db/./greenplum_path.sh; pg_basebackup -c fast -D /data/greenplum/data/master/gpseg-1 -h pc3 -p 5432 --xlog --force-overwrite --write-recovery-conf --target-gp-dbid 7 -E ./db_dumps -E ./gpperfmon/data -E ./gpperfmon/logs -E ./promote --progress --verbose"' cmd had rc=1 completed=True halted=False

请确认原主库的防火墙端口是否开通

image-20210311163518327

45d2e029-d13b-4f8b-9106-e00f05de2a9e

/dev/vdb1 /data xfs nodev,noatime,nobarrier,inode64 0 0

docker run -d -p 5432:5432 lyasper/greenplum:6

gpdeletesystem -d /data/master/gpseg-1 -f

标签：aivmp,安装,GreenPlum,greenplum,master,home,data,gpadmin
From： https://www.cnblogs.com/edclol/p/17282478.html

相关文章

  • akshare库安装及使用
    一：背景作为一个996的加班狗，自己维护的股票分析系统的来源经常进行反爬处理或者干脆停止维护了，换一个来源的话，格式分析脚本也经常需要更换，经过百度之后，发现有大佬维护了一个akshare库，可以获取大量金融相关的数据，并且转换为统一的格式，然后基于从akshare库返回的数据进行加工分......
  • StreamSets单机版安装文档
    StreamSets单机版安装文档解压安装包到opt目录tar-zxvfstreamsets-datacollector-common-3.22.3.tgz-C/opt修改为sdc配置为表单认证cd/opt/streamsets-datacollector-3.22.3/vietc/sdc.propertieshttp.authentication=form后台启动nohup./bin/streamsets......
  • Kerberos 安装
    Kerberos安装1.Kerberos服务端安装服务端重点三个配置文件：/etc/krb5.conf/var/kerberos/krb5kdc/kdc.conf/var/kerberos/krb5kdc/kadm5.acl注意点：防止配置文件格式错误，如编辑过程中导致内容缺失。查询软件对加密算法的支持程度，如降低版本hadoop需要去除掉aes和......
  • npm安装
    npm&&NVM安装nvm安装NVM_DIR="/root/.nvm"NODE_VERSION=17.9.0curl-o-https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh|bash\&&[-s"$NVM_DIR/nvm.sh"]&&\."$NVM_DIR/nvm.sh"\&......
  • Redis6 集群单机安装
    Redis6集群单机安装官网下载https://download.redis.io/releases/redis-6.2.6.tar.gzhttp://download.redis.io/redis-stable.tar.gz文档参考https://redis.io/documentation解压编译#安装编译需要的tclyum-yinstallgcctclrm-rf/data/redismkdir-p/data/re......
  • 【HMS Core】您的设备未安装最新版本的HMS Core，XXX无法正常运行
    【关键字】未安装HMSCore、XXX无法正常运行、IAP应用内支付 【问题描述】项目中接入了华为应用内支付SDK，在使用时调用了如下接口：Iap.getIapClient(activity).isEnvReady然后弹出了一个对话框，提示：“您的设备未安装最新版本的HMSCore，XXX无法正常运行”，详情跟下图中的弹框是......
  • docker 容器内安装ps命令
    转载自：https://blog.csdn.net/HXNLYW/article/details/88127414============= dockerexec-it793d8/bin/sh进入容器内，使用ps-ef|grepjava报了错： 问题出在docker容器linux没有安装procps工具在命令行中输入：apt-getupdate&&apt-getinstallprocps 敲下回......
  • kafka的安装与配置
    安装并配置KafkaBroker这里将kafka和zookeeper部署在一起了。1、下载与安装Kafkakafka官网https://kafka.apache.org/downloads所以这里推荐的版本是kafka_2.12-2.7.2.tgz将下载下来的安装包直接解压到一个路径下即可完成kafka的安装，这里统一将kafka安装到/usr/local目录......
  • 安装node-sass
    报错npmERR!code1npmERR!pathnode_modules\node-sassnpmERR!commandfailednpmERR!commandcnodescripts/build.js安装镜像源：npmconfigsetregistryhttps://registry.npm.taobao.org/配置变量指向淘宝镜像：npmconfigsetsass_binary_sitehttps://np......
  • Redis下载与安装
    在Linux系统安装Redis步骤：1、将Redis安装包上传到Linux2、解压安装包，命令：tar-zxvfredis-4.0.0.tar.gz-C/usr/local3、安装Redis的依赖环境gcc，命令：yuminstallgcc-c++4、进入/usr/local/redis-4.0.0，进行编译，命令：make5、进入redis的src目录，进行安装，命令：makeinstall 在......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99