首页 > 其他分享 >高可用 ZooKeeper 集群部署（进阶中级）

高可用 ZooKeeper 集群部署（进阶中级）

时间：2023-07-06 18:57:57浏览次数：33  
标签：src 进阶 root ZooKeeper hadoop 集群 usr master local

高可用 ZooKeeper 集群部署（进阶中级）

免密登录

[root@master ~]# ssh-keygen
[root@master ~]# ssh-copy-id root@slave1
[root@master ~]# ssh-copy-id root@slave2
[root@master ~]# ssh slave1
Last login: Tue May 23 22:10:27 2023 from 192.168.100.1
[root@slave1 ~]# exit
logout
Connection to slave1 closed.
[root@master ~]#

1. 实验任务一：ZooKeeper 安装部署

1.1. 步骤一：安装 ZooKeeper
[root@master src]# cd
[root@master ~]# ls
anaconda-ks.cfg  zookeeper-3.4.8.tar.gz
[root@master ~]# tar xf zookeeper-3.4.8.tar.gz -C /usr/local/src/
[root@master ~]# cd /usr/local/src/
[root@master src]# ls
hadoop  jdk  zookeeper-3.4.8
[root@master src]# mv zookeeper-3.4.8/ zookeeper
[root@master src]# ls
hadoop  jdk  zookeeper
1.2.步骤二：创建 ZooKeeper 数据目录
[root@master src]# cd zookeeper/
[root@master zookeeper]# ls
bin          contrib          ivy.xml      README_packaging.txt  zookeeper-3.4.8.jar
build.xml    dist-maven       lib          README.txt            zookeeper-3.4.8.jar.asc
CHANGES.txt  docs             LICENSE.txt  recipes               zookeeper-3.4.8.jar.md5
conf         ivysettings.xml  NOTICE.txt   src                   zookeeper-3.4.8.jar.sha1
[root@master zookeeper]# mkdir data
[root@master zookeeper]# mkdir logs
[root@master zookeeper]# ls
bin          data             lib                   README.txt               zookeeper-3.4.8.jar.md5
build.xml    dist-maven       LICENSE.txt           recipes                  zookeeper-3.4.8.jar.sha1
CHANGES.txt  docs             logs                  src
conf         ivysettings.xml  NOTICE.txt            zookeeper-3.4.8.jar
contrib      ivy.xml          README_packaging.txt  zookeeper-3.4.8.jar.asc

2.实验任务二：ZooKeeper 文件参数配置

2.1. 步骤一：配置 ZooKeeper 环境变量
[root@master ~]# vi /etc/profile.d/zookeeper.sh

export JAVA_HOME=/usr/local/src/java 
export PATH=$PATH:$JAVA_HOME/bin
export ZK_HOME=/usr/local/src/zookeeper
export PATH=$PATH:$ZK_HOME/bin
2.2. 步骤二：修改 zoo.cfg 配置文件
[root@master ~]# cd /usr/local/src/zookeeper/conf
[root@master conf]# ls
configuration.xsl  log4j.properties  zoo_sample.cfg
[root@master conf]# cp zoo_sample.cfg zoo.cfg
[root@master conf]# vi zoo.cfg 
#添加并更改如下配置：
#修改
dataDir=/usr/local/src/zookeeper/data
#增加
dataLogDir=/usr/local/src/zookeeper/logs
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888
2.3. 步骤三：创建 myid 配置文件
[root@master conf]# cd ..
[root@master zookeeper]# cd data/
[root@master data]# echo "1" > myid
[root@master data]# cat myid
1

3. 实验任务三：ZooKeeper 集群启动

3.1. 步骤一：分发 ZooKeeper 集群
[root@master ~]# scp -r /usr/local/src/zookeeper/ root@slave1:/usr/local/src/
[root@master ~]# scp -r /usr/local/src/zookeeper/ root@slave2:/usr/local/src/
#分发环境变量并使其生效
[root@master ~]# scp /etc/profile.d/zookeeper.sh root@slave1:/etc/profile.d/
zookeeper.sh                                                             100%  141   103.4KB/s   00:00    
[root@master ~]# scp /etc/profile.d/zookeeper.sh root@slave2:/etc/profile.d/
zookeeper.sh                                                             100%  141    38.1KB/s   00:00
3.2. 步骤二：修改 myid 配置
[root@master zookeeper]# cd data
[root@master data]# ls
myid
[root@master data]# cat myid
1
[root@slave1 ~]# cd /usr/local/src/zookeeper/data/
[root@slave1 data]# echo "2" > myid
[root@slave1 data]# cat myid 
2
[root@slave2 ~]# cd /usr/local/src/zookeeper/data/
[root@slave2 data]# echo "3" > myid
[root@slave2 data]# cat myid 
3
3.3. 步骤三：修改 ZooKeeper 安装目录的归属用户为 hadoop 用户。
[root@master data]# # chown -R hadoop:hadoop /usr/local/src/zookeeper
[root@slave1 data]# # chown -R hadoop:hadoop /usr/local/src/zookeepe
[root@slave2 data]# # chown -R hadoop:hadoop /usr/local/src/zookeepe
3.4. 步骤四：启动 ZooKeeper 集群
关闭防火墙
[root@master data]# systemctl stop firewalld.service
[root@slave1 data]# systemctl stop firewalld.service
[root@slave2 data]# systemctl stop firewalld.service
关闭防火墙自启
[root@master data]# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@slave1 data]# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@slave2 data]# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

#同时启动三个节点的 zookeeper
[hadoop@master ~]$ zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[hadoop@master ~]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: follower

[hadoop@slave1 ~]$ zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[hadoop@slave1 ~]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: leader

[hadoop@slave2 ~]$ zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[hadoop@slave2 ~]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: follower

2. 实验二 Hadoop HA 集群部署

1.实验任务一：Hadoop HA 文件参数配置

1.1. 步骤一：解压安装 Hadoop
[root@master ~]# ls
anaconda-ks.cfg  hadoop-2.7.1.tar.gz  zookeeper-3.4.8.tar.gz
[root@master ~]# tar xf hadoop-2.7.1.tar.gz -C /usr/local/src/
1.2步骤二：更改 hadoop 文件名
[root@master src]# mv hadoop-2.7.1/ hadoop
[root@master src]# ls
hadoop  jdk  zookeeper
1.3. 步骤三：配置 hadoop 环境变量
[root@master src]# vi /etc/profile.d/hadoop.sh
export HADOOP_HOME=/usr/local/src/hadoop 
export HADOOP_PREFIX=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export JAVA_HOME=/usr/local/src/java 
export PATH=$PATH:$JAVA_HOME/bin
export ZK_HOME=/usr/local/src/zookeeper
export PATH=$PATH:$ZK_HOME/bin
1.4.步骤四：配置 hadoop-env.sh 配置文件
[root@master src]# cd hadoop/etc/hadoop/
[root@master hadoop]# vi hadoop-env.sh 
#在最下面添加如下配置：
export JAVA_HOME=/usr/local/src/java
1.5. 步骤五：配置 core-site.xml 配置文件
[root@master hadoop]# vi core-site.xml

<!-- 指定 hdfs 的 nameservice 为 mycluster -->
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://mycluster</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>file:/usr/local/src/hadoop/tmp</value>
</property>
<!-- 指定 zookeeper 地址 -->
<property>
    <name>ha.zookeeper.quorum</name>
    <value>master:2181,slave1:2181,slave2:2181</value>
</property>
<!-- hadoop 链接 zookeeper 的超时时长设置 -->
<property>
    <name>ha.zookeeper.session-timeout.ms</name>
    <value>30000</value>
    <description>ms</description>
</property>
<property>
    <name>fs.trash.interval</name>
    <value>1440</value>
</property>
1.6. 步骤六：配置 hdfs-site.xml 配置文件
[root@master hadoop]# vi hdfs-site.xml 
< property >
    < name > dfs.qjournal.start - segment.timeout.ms < / name >
    < value > 60000 < / value >
< / property >
< property >
    < name > dfs.nameservices < / name >
    < value > mycluster < / value >
< / property >
< property >
    < name > dfs.ha.namenodes.mycluster < / name >
    < value > master, slave1 < / value >
< / property >
< property >
    < name > dfs.namenode.rpc - address.mycluster.master < / name >
    < value > master: 8020 < / value >
< / property >
< property >
    < name > dfs.namenode.rpc - address.mycluster.slave1 < / name >
    < value > slave1: 8020 < / value >
< / property >
< property >
    < name > dfs.namenode.http - address.mycluster.master < / name >
    < value > master: 50070 < / value >
< / property >
< property >
    < name > dfs.namenode.http - address.mycluster.slave1 < / name >
    < value > slave1: 50070 < / value >
< / property >
< property >
    < name > dfs.namenode.shared.edits.dir < / name >
    < value > qjournal: // master: 8485;slave1: 8485;slave2: 8485 / mycluster < / value >
< / property >
< property >
    < name > dfs.client.failover.proxy.provider.mycluster < / name >
    < value > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider < / value >
< / property >
< property >
    < name > dfs.ha.fencing.methods < / name >
    < value >
    sshfence
    shell( / bin / true)
    < / value >
< / property >
< property >
    < name > dfs.permissions.enabled < / name >
    < value > false < / value >
< / property >
< property >
    < name > dfs.support.append < / name >
    < value > true < / value >
< / property >
< property >
    < name > dfs.ha.fencing.ssh.private - key - files < / name >
    < value > / root /.ssh / id_rsa < / value >
< / property >
< property >
    < name > dfs.replication < / name >
    < value > 2 < / value >
< / property >
< property >
    < name > dfs.namenode.name.dir < / name >
    < value > / usr / local / src / hadoop / tmp / hdfs / nn < / value >
< / property >
< property >
    < name > dfs.datanode.data.dir < / name >
    < value > / usr / local / src / hadoop / tmp / hdfs / dn < / value >
< / property >
< property >
    < name > dfs.journalnode.edits.dir < / name >
    < value > / usr / local / src / hadoop / tmp / hdfs / jn < / value >
< / property >
< property >
    < name > dfs.ha.automatic - failover.enabled < / name >
    < value > true < / value >
< / property >
< property >
    < name > dfs.webhdfs.enabled < / name >
    < value > true < / value >
< / property >
< property >
    < name > dfs.ha.fencing.ssh.connect - timeout < / name >
    < value > 30000 < / value >
< / property >
< property >
    < name > ha.failover - controller.cli - check.rpc - timeout.ms < / name >
    < value > 60000 < / value >
< / property >
1.7. 步骤七：配置 mapred-site.xml 配置文件
[root@master hadoop]# cp mapred-site.xml.template mapred-site.xml
[root@master hadoop]# vi mapred-site.xml

<!-- 指定 mr 框架为 yarn 方式 -->
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<!-- 指定 mapreduce jobhistory 地址 -->
<property>
    <name>mapreduce.jobhistory.address</name>
    <value>master:10020</value>
</property>
<!-- 任务历史服务器的 web 地址 -->
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>master:19888</value>
</property>
1.8. 步骤八：配置 yarn-site.xml 配置文件
[root@master hadoop]# vi yarn-site.xml

<!-- Site specific YARN configuration properties -->
<!-- 开启 RM 高可用 -->
<property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
</property>
<!-- 指定 RM 的 cluster id -->
<property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>yrc</value>
</property>
<!-- 指定 RM 的名字 -->
<property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
</property>
<!-- 分别指定 RM 的地址 -->
<property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>master</value>
</property>
<property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>slave1</value>
</property>
<!-- 指定 zk 集群地址 -->
<property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>master:2181,slave1:2181,slave2:2181</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>86400</value>
</property>
<!-- 启用自动恢复 -->
<property>
    <name>yarn.resourcemanager.recovery.enabled</name>
    <value>true</value>
</property>
<!-- 制定 resourcemanager 的状态信息存储在 zookeeper 集群上 -->
<property>
    <name>yarn.resourcemanager.store.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
1.9. 步骤九：配置 slaves 配置文件
[root@master hadoop]# vi slaves

master
slave1
slave2
1.10. 步骤十：解压包到指定目录

namenode、datanode、journalnode 等存放数据的公共目录为/usr/local/src/hadoop/tmp；

[root@master hadoop]# mkdir -p /usr/local/src/hadoop/tmp/hdfs/nn
[root@master hadoop]# mkdir -p /usr/local/src/hadoop/tmp/hdfs/dn
[root@master hadoop]# mkdir -p /usr/local/src/hadoop/tmp/hdfs/jn
[root@master hadoop]# mkdir -p /usr/local/src/hadoop/tmp/logs
1.11. 步骤十一：分发文件
[root@master hadoop]# scp -r /etc/profile.d/hadoop.sh root@slave1:/etc/profile.d/
hadoop.sh                                        100%  602    41.7KB/s   00:00    
[root@master hadoop]# scp -r /etc/profile.d/hadoop.sh root@slave2:/etc/profile.d/
hadoop.sh                                        100%  602   517.3KB/s   00:00  
[root@master hadoop]# scp -r /usr/local/src/hadoop root@slave1:/usr/local/src/
[root@master hadoop]# scp -r /usr/local/src/hadoop root@slave2:/usr/local/src/
1.12. 步骤十二：修改目录所有者和所有者组
[root@master ~]# chown -R hadoop:hadoop /usr/local/src/hadoop/ 
[root@slave1 ~]# chown -R hadoop:hadoop /usr/local/src/hadoop/
[root@slave2 ~]# chown -R hadoop:hadoop /usr/local/src/hadoop/

实验一：高可用集群启动

实验任务一：HA 的启动

1.1. 步骤一：启动 journalnode 守护进程
[hadoop@master ~]$ hadoop-daemons.sh start journalnode
master: starting journalnode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-journalnode-master.out
slave1: starting journalnode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-journalnode-slave1.out
slave2: starting journalnode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-journalnode-slave2.out
1.2. 步骤二：初始化 namenode
[hadoop@master ~]$ hdfs namenode -format
23/05/28 08:14:19 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.100.10
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.1
STARTUP_MSG:   classpath = /usr/local/src/hadoop/etc/hadoop:/usr/local/src/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/src/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/src/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/src/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/src/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/src/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/src/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-cli-
.....
retain 1 images with txid >= 0
23/05/28 08:14:41 INFO util.ExitUtil: Exiting with status 0
23/05/28 08:14:41 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.100.10
************************************************************/
1.3. 步骤三：注册 ZNode
[hadoop@master ~]$ hdfs zkfc -formatZK
23/05/28 08:16:36 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at master/192.168.100.10:8020
23/05/28 08:16:36 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
23/05/28 08:16:36 INFO zookeeper.ZooKeeper: Client environment:host.name=master
23/05/28 08:16:36 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_152
23/05/28 08:16:36 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
23/05/28 08:16:36 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/src/jdk/jre
23/05/28 08:16:36 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/usr/local/src/hadoop/etc/hadoop:/usr/local/src/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/src/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/src/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/src/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/src/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/src/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/src/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/src/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/src/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/src/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/src/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/src/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/src/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-collections-3.2.1
......

1.4. 步骤四：启动 hdfs

[hadoop@master ~]$  start-dfs.sh
Starting namenodes on [slave1 master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.out
slave1: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-slave1.out
slave2: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.out
master: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-master.out
slave1: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.out
Starting journal nodes [master slave1 slave2]
master: journalnode running as process 1461. Stop it first.
slave2: journalnode running as process 1269. Stop it first.
slave1: journalnode running as process 1259. Stop it first.
Starting ZK Failover Controllers on NN hosts [slave1 master]
master: starting zkfc, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-zkfc-master.out
slave1: starting zkfc, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-zkfc-slave1.out
1.5. 步骤五：启动 yarn
[hadoop@master ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.out
slave1: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.out
slave2: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.out
master: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-master.out
1.6. 步骤六：同步 master 数据
复制 namenode 元数据到其它节点(在 master 节点执行)
[hadoop@master ~]$ scp -r /usr/local/src/hadoop/tmp/hdfs/nn/* slave1:/usr/local/src/hadoop/tmp/hdfs/nn/
VERSION                                          100%  206   313.1KB/s   00:00    
seen_txid                                        100%    2     4.3KB/s   00:00    
fsimage_0000000000000000000.md5                  100%   62   157.1KB/s   00:00    
fsimage_0000000000000000000                      100%  353   719.7KB/s   00:00    
in_use.lock                                      100%   11    21.6KB/s   00:00    
[hadoop@master ~]$ scp -r /usr/local/src/hadoop/tmp/hdfs/nn/* slave2:/usr/local/src/hadoop/tmp/hdfs/nn/
VERSION                                          100%  206   104.4KB/s   00:00    
seen_txid                                        100%    2     2.0KB/s   00:00    
fsimage_0000000000000000000.md5                  100%   62    65.8KB/s   00:00    
fsimage_0000000000000000000                      100%  353   334.5KB/s   00:00    
in_use.lock                                      100%   11     9.2KB/s   00:00
1.7. 步骤七：在 slave1 上启动 resourcemanager 和 namenode 进程
[hadoop@slave1 ~]$  yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-slave1.out
[hadoop@slave1 ~]$ hadoop-daemon.sh start namenode
namenode running as process 1345. Stop it first.
1.8. 步骤九：启动 MapReduce 任务历史服务器
[hadoop@master ~]$  yarn-daemon.sh start proxyserver
starting proxyserver, logging to /usr/local/src/hadoop/logs/yarn-hadoop-proxyserver-master.out
[hadoop@master ~]$ jps
2177 DFSZKFailoverController
2306 ResourceManager
1460 JournalNode
2421 NodeManager
1734 NameNode
2810 Jps
1339 QuorumPeerMain
1871 DataNode
[hadoop@master ~]$ mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /usr/local/src/hadoop/logs/mapred-hadoop-historyserver-master.out
1.9. 步骤十：查看端口和进程
#master
[hadoop@master ~]$ jps
2177 DFSZKFailoverController
2849 JobHistoryServer
2306 ResourceManager
1460 JournalNode
2421 NodeManager
1734 NameNode
1339 QuorumPeerMain
2893 Jps
1871 DataNode
#slave1
[hadoop@slave1 ~]$ jps
1411 JournalNode
1317 QuorumPeerMain
2552 Jps
2473 NameNode
1740 DFSZKFailoverController
2351 ResourceManager
2451 NodeManager
1891 DataNode
#slave2
[hadoop@slave2 ~]$ jps
1252 QuorumPeerMain
1576 NodeManager
1434 DataNode
1340 JournalNode
1708 Jps

master:50070

image-20230531111456429

slave1:50070

image-20230531111521204

master:8088

image-20230531111648126

2.实验任务二：HA 的测试

2.1. 步骤一：创建一个测试文件
[hadoop@master ~]$ vi y.txt

Hello World
Hello Hadoop
.2.2. 步骤二：在 hdfs 创建文件夹
[hadoop@master ~]$ hadoop fs -mkdir /input
.2.3. 步骤三：将 a.txt 传输到 input 上
[hadoop@master ~]$ hadoop fs -put ~/a.txt /input
2.4. 步骤四：进入到 jar 包测试文件目录下
[hadoop@master ~]$ cd /usr/local/src/hadoop/share/hadoop/mapreduce/
2.5. 步骤五：测试 mapreduce
[hadoop@master mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.7.1.jar
wordcount /input/wbb.txt /output
23/05/30 07:08:46 INFO input.FileInputFormat: Total input paths to process : 1
23/05/30 07:08:46 INFO mapreduce.JobSubmitter: number of splits:1
23/05/30 07:08:46 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1685444184808_0001
23/05/30 07:08:47 INFO impl.YarnClientImpl: Submitted application
application_1685444184808_0001
23/05/30 07:08:47 INFO mapreduce.Job: The url to track the job:
http://master:8088/proxy/application_1685444184808_0001/
23/05/30 07:08:47 INFO mapreduce.Job: Running job: job_1685444184808_0001
23/05/30 07:08:59 INFO mapreduce.Job: Job job_1685444184808_0001 running in uber
mode : false
23/05/30 07:08:59 INFO mapreduce.Job: map 0% reduce 0%
23/05/30 07:09:07 INFO mapreduce.Job: map 100% reduce 0%
23/05/30 07:09:11 INFO mapreduce.Job: map 100% reduce 100%
23/05/30 07:09:12 INFO mapreduce.Job: Job job_1685444184808_0001 completed
successfully
23/05/30 07:09:12 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=43
FILE: Number of bytes written=236319
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=120
HDFS: Number of bytes written=25
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=6390
Total time spent by all reduces in occupied slots (ms)=1593
Total time spent by all map tasks (ms)=6390
Total time spent by all reduce tasks (ms)=1593
Total vcore-seconds taken by all map tasks=6390
Total vcore-seconds taken by all reduce tasks=1593
Total megabyte-seconds taken by all map tasks=6543360
Total megabyte-seconds taken by all reduce tasks=1631232
Map-Reduce Framework
Map input records=2
Map output records=4
Map output bytes=41
Map output materialized bytes=43
Input split bytes=95
Combine input records=4
Combine output records=3
Reduce input groups=3
高可用性验证
Reduce shuffle bytes=43
Reduce input records=3
Reduce output records=3
Spilled Records=6
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=404
CPU time spent (ms)=2500
Physical memory (bytes) snapshot=433823744
Virtual memory (bytes) snapshot=4241760256
Total committed heap usage (bytes)=283115520
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=25
File Output Format Counters
Bytes Written=25
2.7. 步骤七：查看文件测试的结果
[hadoop@master mapreduce]$ hadoop fs -cat /output/part-r-00000
Hadoop 1
Hello 2
World 1

3. 实验任务三：高可用性验证

3.1. 步骤一：:自动切换服务状态
[hadoop@master ~]$ hdfs haadmin -getServiceState slave1
active
[hadoop@master ~]$ hdfs haadmin -getServiceState master
standby
3.2. 步骤二：手动切换服务状态
[hadoop@slave1 ~]$ hadoop-daemon.sh stop namenode
stopping namenode
[hadoop@master ~]$  hdfs haadmin -getServiceState master
active
[hadoop@master ~]$ hdfs haadmin -getServiceState slave1
23/05/30 23:35:24 INFO ipc.Client: Retrying connect to server: slave1/192.168.100.20:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From master/192.168.100.10 to slave1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

查看 web 服务端

master:50070

image-20230531113653949

slave1:50070

image-20230531113741536

标签：src,进阶,root,ZooKeeper,hadoop,集群,usr,master,local
From： https://www.cnblogs.com/shuangmu668/p/17533034.html

相关文章

  • Hive 组件（进阶中级）
    Hive组件（进阶中级）1.实验一：Hive组件部署1.实验任务一：环境需求1.1.步骤一：解压安装文件[root@master~]#lsanaconda-ks.cfgjdk-8u152-linux-x64.tar.gzapache-hive-2.0.0-bin.tar.gzmysql-5.7.18.ziphadoop-2.7.1.tar.gzstudent.csvhbase-......
  • HBase 分布式部署（进阶中级）
    1.HBase分布式部署（进阶中级）1.实验任务一：部署前期准备1.1.步骤一：安装部署hadoopha分布式环境1.2.步骤二：解压安装文件[root@master~]#cd[root@master~]#lsanaconda-ks.cfgjdk-8u152-linux-x64.tar.gzhadoop-2.7.1.tar.gzzookeeper-3.4.8.tar.gzhb......
  • 选对方法，K8s 多集群管理没那么难
    作者：庄宇Kubernetes作为一项核心技术已成为现代应用程序架构的基础，将Kubernetes作为容器编排系统已发展为越来越多企业的必然选择。随着对云计算接受程度不断提高，以及企业规模和业务持续发展的共同驱动下，越来越多的企业在考虑或已经采用多云和混合云方案，以提升架构的灵活性和......
  • 2. HTML 进阶之HTTP请求
    1)请求组成请求由三部分组成请求行请求头请求体可以用telnet程序测试2)请求方式与数据格式get请求示例GET/test2?name=%E5%BC%A0&age=20HTTP/1.1Host:localhost%E5%BC%A0是【张】经过URL编码后的结果post请求示例POST/test2HTTP/1.1Host:localho......
  • springcloud - zookeeper的基本配置和使用
     第一步：导入依赖     <dependency>       <groupId>org.springframework.cloud</groupId>       <artifactId>spring-cloud-starter-zookeeper-discovery</artifactId>       <!--排除启动器中配置好的版本-->    ......
  • Docker+Emqx+Nginx集群及负载均衡的搭建 MQTT服务器
    Docker+Emqx+Nginx集群及负载均衡的搭建MQTT服务器1.拉取镜像dockerpullemqx/emqx2.创建虚拟网络#-d参数默认为bridge，可省略dockernetworkcreate-dbridge--subnet=172.18.0.0/16emqx_bridge3.创建Docker容器#节点1dockerrun-d--hostnameemqx01--nameem......
  • 记一次zookeeper占满磁盘的故障
    今天有测试找到我说zookeeper161主机满了，走到主机查看发现确实满了，但是我们的log也已经清了，但是data文件里边全是快照文件。这里就要讲一讲zookeeper文件的删除了，首先我们不能贸然用rm命令删除快照文件，如果直接删完会导致丢失好多数据，所以我们需要设置相关参数，并使用一些优雅的......
  • 如何实现基于kubernetes安装和运维大数据集群的具体操作步骤
    基于Kubernetes安装和运维大数据集群介绍Kubernetes是一个开源的容器编排平台，可以帮助我们管理和运行容器化的应用程序。它提供了许多强大的功能，使得在大数据环境中安装和运维大数据集群变得更加容易。在本文中，我们将介绍如何使用Kubernetes来安装和运维一个大数据集群。我们将......
  • ZLmediakit集群部署
    1简单理解2如何简单部署测试2.1:我在10.1.1.1的机器上有拉IPC-A摄像头的数据源；这个摄像头通过部署在10.1.1.1上的ZLM去拉，转协议RTSP的地址是rtsp://10.1.1.1:554/rtp/stream_12.2：我本地（127.0.0.1）启动ZLM服务，通过VLC拉stream1：此时肯定是找不到的，因为我本地就没有叫stream_1......
  • java进阶3：GC 的背景与一般原理
    为什么会有GC其最本质的原因是因为内存资源的稀缺性。我们计算机最核心的资源是CPU和内存，CPU是随着计算机一直存在的东西，核数有限但是一直存在；但内存比较稀缺，A占满了，B就不能用了，我们怎么可以共享使用这个内存呢，这就是GC产生的原因了。背景：在传统的编程语言中，程序员需要手动分配和释......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99