首页 > 其他分享 >Hadoop 集群配置

Hadoop 集群配置

时间:2022-10-18 21:00:44浏览次数:69  
标签:node01 配置 Hadoop dfs hadoop etc 集群 yarn site

参考官网整理可能的配置:https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-common/ClusterSetup.html

etc/hadoop/core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>node01:50070</value>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>131072</value>
  </property>
</configuration>

etc/hadoop/hdfs-site.xml

<configuration>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/tmp/dfs/namenode/name/</value>
  </property>
  <property>
    <name>dfs.blocksize</name>
    <value>268435456</value>
  </property>
  <property>
    <name>dfs.namenode.handler.count</name>
    <value>10</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/tmp/dfs/datanode/data/</value>
  </property>
</configuration>

etc/hadoop/yarn-site.xml

<configuration>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>86400</value>
  </property>
  <property>
    <name>yarn.log-aggregation.retain-check-interval-seconds</name>
    <value>-1</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>node01</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>128</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>256</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>256</value>
  </property>
</configuration>

etc/hadoop/mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>node01:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>19888</value>
  </property>
</configuration>

etc/hadoop/slaves

node01
node02
node03

标签:node01,配置,Hadoop,dfs,hadoop,etc,集群,yarn,site
From: https://www.cnblogs.com/zby9527/p/16804109.html

相关文章