-
按照此文档操作,可以一次部署成功:Ubutnu 20.04 安装和使用单机版hadoop 3.2
-
部署之后,提交测试任务报资源问题。原因是yarn还需要配置,如下:
$ cat yarn-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- 指定MR走shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定ResourceManager的地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>192.168.122.200</value>
</property>
<!-- 环境变量的继承 -->
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
- 示例:
$ hadoop jar ~/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar wordcount /1 /2 /out3
2023-01-06 10:23:58,058 INFO client.RMProxy: Connecting to ResourceManager at /192.168.122.200:8032
2023-01-06 10:23:58,362 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1672999900900_0002
2023-01-06 10:23:58,508 INFO input.FileInputFormat: Total input files to process : 2
2023-01-06 10:23:58,557 INFO mapreduce.JobSubmitter: number of splits:2
2023-01-06 10:23:58,701 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1672999900900_0002
2023-01-06 10:23:58,702 INFO mapreduce.JobSubmitter: Executing with tokens: []
2023-01-06 10:23:58,796 INFO conf.Configuration: resource-types.xml not found
2023-01-06 10:23:58,796 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-01-06 10:23:58,829 INFO impl.YarnClientImpl: Submitted application application_1672999900900_0002
2023-01-06 10:23:58,849 INFO mapreduce.Job: The url to track the job: http://192.168.122.200:8088/proxy/application_1672999900900_0002/
2023-01-06 10:23:58,849 INFO mapreduce.Job: Running job: job_1672999900900_0002
2023-01-06 10:24:03,962 INFO mapreduce.Job: Job job_1672999900900_0002 running in uber mode : false
2023-01-06 10:24:03,965 INFO mapreduce.Job: map 0% reduce 0%
2023-01-06 10:24:08,059 INFO mapreduce.Job: map 100% reduce 0%
2023-01-06 10:24:11,079 INFO mapreduce.Job: map 100% reduce 100%
2023-01-06 10:24:12,116 INFO mapreduce.Job: Job job_1672999900900_0002 completed successfully
...
标签:INFO,10,01,06,hadoop,mapreduce,3.2,2023,单机版
From: https://www.cnblogs.com/dewan/p/17031317.html