首页 > 其他分享 >hadoop HA高可用集群搭建

hadoop HA高可用集群搭建

时间：2023-09-28 21:00:45浏览次数：37  
标签：-- hadoop yarn 集群 dfs billsaifu namenode HA

首先是四项配置

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://mycluster</value>
    </property>
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>billsaifu:2181,hadoop1:2181,hadoop2:2181</value>
    </property>
<!--    <property>-->
<!--        <name>fs.defaultFS</name>-->
<!--        <value>hdfs://billsaifu:9000</value>-->
<!--    </property>-->
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>bill</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop-3.2.4/tmp</value>
    </property>
    <property>

        <name>dfs.datanode.max.xcievers</name>

        <value>4096</value>

        <dedication> Datanode 有一个同时处理文件的上限，至少要有4096</dedication>

    </property>
    <property>
        <name>hadoop.proxyuser.bill.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.bill.hosts</name>
        <value>*</value>
    </property>

    <property>
        <name>hadoop.proxyuser.bill.users</name>
        <value>*</value>
    </property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.replication.min</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.nameservices</name>
        <value>mycluster</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.mycluster</name>
        <value>nn1,nn2</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn1</name>
        <value>billsaifu:9000</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn1</name>
        <value>billsaifu:9870</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn2</name>
        <value>hadoop2:9000</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn2</name>
        <value>hadoop2:9870</value>
    </property>
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://billsaifu:8485;hadoop1:8485;hadoop2:8485/mycluster</value>
    </property>
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/mnt/hadoop_data/journal_data</value>
    </property>
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.mycluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>
            sshfence
        </value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/bill/.ssh/id_rsa</value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>30000</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/mnt/hadoop_data/hadoop_nndata</value>
    </property>
    <property>
        <name>dfs.namenode.handler.count</name>
        <value>100</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/mnt/hadoop_data/hdfs_dndata</value>
    </property>
    <property>
        <name>dfs.permissions.superusergroup</name>
        <value>freedom</value>
    </property>

    <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>true</value>
    </property>

    <property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.namenode.clusterID</name>
        <value>hadoopMaster</value>
    </property>
<!--    &lt;!&ndash; nn web 端访问地址&ndash;&gt;-->
<!--    <property>-->
<!--        <name>dfs.namenode.http-address</name>-->
<!--        <value>billsaifu:9870</value>-->
<!--    </property>-->

<!--    &lt;!&ndash; 2nn web 端访问地址&ndash;&gt;-->
<!--    <property>-->
<!--        <name>dfs.namenode.secondary.http-address</name>-->
<!--        <value>hadoop2:9868</value>-->
<!--    </property>-->

</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx1096M</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>mapreduce.reduce.java.opts</name>
        <value>-Xmx2560M</value>
    </property>
    <property>
        <name>mapreduce.task.io.sort.mb</name>
        <value>512</value>
    </property>
    <property>
        <name>mapreduce.task.io.sort.factor</name>
        <value>100</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hdfs://billsaifu:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>http://billsaifu:19888</value>
    </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/mnt/hadoop_data/yarn_nmdata</value>
    </property>

    <property>
        　　<name>yarn.nodemanager.resource.memory-mb</name>
        　　<value>2048</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.QUEUE_CRA.capacity</name>
        <value>50</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.QUEUE_CRA.maximum-capacity</name>
        <value>100</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.QUEUE_CRA.user-limit-factor</name>
        <value>1</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.QUEUE_CRA.state</name>
        <value>RUNNING</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log.server.url</name>
        <value>http://billsaifu:19889/jobhistory/logs</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>yrc</value>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop1</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop2</value>
    </property>
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>billsaifu:2181,hadoop1:2181,hadoop2:2181</value>
    </property>
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>

</configuration>

启动journalnode

billsaifu:hadoop-daemon.sh start journalnode

hadoop1:hadoop-daemon.sh start journalnode

hadoop2:hadoop-daemon.sh start journalnode

格式化namenode

hadoop namenode -format

hadoop2上同步billsaifu上的节点的数据

hdfs namenode -bootstrapStandBy

初始化zookeeper

hdfs zkfc -formatZK

start-all.sh启动集群

还有一件事情需要注意的是

因为之前我搭建了传统的hadoop集群因此当我初始化namenode后，datanode中的namenode是原来的那个版本需要修改过来不然无法启动datanode

进入hadoop_nndate下的VERSION

复制这个ID

clusterID=CID-9785fc23-8cd5-41f6-80ca-b97c84335afa

 替换datanode的VESION的ID

 

标签：--,hadoop,yarn,集群,dfs,billsaifu,namenode,HA
From： https://www.cnblogs.com/liyiyang/p/17736479.html

相关文章

  • 使用 Terraform 创建K8S集群
    Terraform是一个开源的基础设施即代码(InfrastructureasCode,IaC)工具，由HashiCorp公司开发和维护。它用于自动化和管理云基础设施、服务和资源的创建、配置和部署。Terraform允许开发人员和运维团队以声明性的语言描述基础设施，并使用代码的方式来管理基础设施，从而提高......
  • 02. Kubeadm部署Kubernetes集群
    目录1、前言2、Kubernetes部署方式3、kubeadmin部署3.1、关闭防火墙3.2、配置阿里云Kubernetes源3.3、安装kubeadm，kubelet，kubectl3.4、初始化master节点3.5、master节点配置kubectl命令行工具3.6、master节点下载flannel网络配置文件3.7、node1和node2节点加入Kubernetes集群3.8、......
  • 文心一言 VS 讯飞星火 VS chatgpt （93）-- 算法导论9.2 1题
    一、用go语言，证明：在RANDOMIZED-SELECT中，对长度为0的数组，不会进行递归调用。文心一言：在Go语言中，我们可以通过实现一个函数来检查数组的长度，如果长度为0，则不进行递归调用。这里是一个简单的示例：packagemainimport("fmt")funcrandomizedSelect(arr[]int,k......
  • 文心一言 VS 讯飞星火 VS chatgpt （92）-- 算法导论8.3 5题
    五、用go语言，在本节给出的第一个卡片排序算法中，为排序d位十进制数，在最坏情况下需要多少轮排序?在最坏情况下，操作员需要记录多少堆卡片?文心一言：你提到的第一个卡片排序算法可能是指的基数排序（RadixSort）。基数排序是一种非比较型整数排序算法，其原理是将整数按位数切割成不同的数......
  • 文心一言 VS 讯飞星火 VS chatgpt （89）-- 算法导论8.3 2题
    二、用go语言，下面的排序算法中哪些是稳定的：插入排序、归并排序、堆排序和快速排序?给出一个能使任何排序算法都稳定的方法。你所给出的方法带来的额外时间和空间开销是多少？文心一言：在Go语言中，以下排序算法是稳定的：插入排序归并排序而以下的排序算法是不稳定的：堆排序快速排序要使任......
  • ChatGLM2
    下载chatglm2-6bprint('开始加载分词器tokenizer...')tokenizer=AutoTokenizer.from_pretrained("THUDM/chatglm2-6b",trust_remote_code=True)print('开始加载语言模型model...')model=AutoModel.from_pretrained("THUDM/chatglm......
  • 玩转Redhat Linux 8.0系列 | 使用BASH SHELL执行命令
    今天继续分享一些RedhatLinux8.0的知识，记得关注，会一直更新~基本命令语法GNUBourne-AgainShell(bash)这一程序可以解读用户键入的命令。当您准备好执行命令时，请按Enter键。在单独的行上键入每个命令。系统会显示命令输出，然后显示下一shell提示符。[user@host]$whoamiuser[u......
  • ChatGPT 重磅更新可进行实时网络搜索；OpenAI 将构建新的“AI 硬件”丨RTE开发者日报 Vo
    开发者朋友们大家好：这里是「RTE开发者日报」，每天和大家一起看新闻、聊八卦。我们的社区编辑团队会整理分享RTE（RealTimeEngagement）领域内「有话题的新闻」、「有态度的观点」、「有意思的数据」、「有思考的文章」、「有看点的会议」，但内容仅代表编辑的个人观点，欢迎大家留......
  • OpenHarmony定义组件重用样式：@Styles装饰器
    OpenHarmony定义组件重用样式：@Styles装饰器如果每个组件的样式都需要单独设置，在开发过程中会出现大量代码在进行重复样式设置，虽然可以复制粘贴，但为了代码简洁性和后续方便维护，我们推出了可以提炼公共样式进行复用的装饰器@Styles。@Styles装饰器可以将多条样式设置提炼成一个方法......
  • What is Lambda?
    根据我的观察，Lambda是一种比较灵活的形式，需要多看几个案例才能明白它。Lambda是一种简化代码的技术手段，主要用于简化匿名实现类，允许把函数作为一个方法的参数传递进方法中。它本身并不会创造出新的概念和功能来。不过现在很多开源项目都使用到这种技术，我的原则是能看懂别人代码即......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99