spark2.4.4安装

1、环境准备：

下载spark-2.4.4-bin-hadoop2.7.tgz

上传安装包到Linux中

解压安装包

tar -zxf spark-2.4.4-bin-hadoop2.7.tgz -C /hadoop/app

2、配置环境

2.1修改hadoop中yarn-site.xml

    <property>
      <name>yarn.nodemanager.pmem-check-enabled</name>
      <value>false</value>
    </property>
    <property>
      <name>yarn.nodemanager.vmem-check-enabled</name>
      <value>false</value>
    </property>
        

将文件同步到其他节点

scp yarn-site,xml node2:/hadoop/app/hadoop-2.7.2/etc/hadoop/

scp yarn-site,xml node3:/hadoop/app/hadoop-2.7.2/etc/hadoop/

2.2修改spark配置文件

cd /hadoop/app/spark-2.4.4-bin-hadoop2.7/conf

vim slaves

 vim spark-env.sh

 同步配置文件到其他节点

scp slaves node2:/hadoop/app/spark-2.4.4-bin-hadoop2.7/conf

scp slaves node3:/hadoop/app/spark-2.4.4-bin-hadoop2.7/conf

scp spark-env.sh node2:/hadoop/app/spark-2.4.4-bin-hadoop2.7/conf

scp spark-env.sh node3:/hadoop/app/spark-2.4.4-bin-hadoop2.7/conf

3.环境验证

3.1启动yarn

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client /hadoop/app/spark-2.4.4-bin-hadoop2.7examples/jars/spark-examples_2.11-2.4.4.jar 100