Arm64v8 cpu + Centos7 aarch64中安装 Ambari 2.7.3 和 HDP 3.1.0

# 下载不存在的资源的方法
使用迅雷云盘，添加下载任务到云盘，有一定的概率下载到已经被删除的资源。 比如下载HDP相关的资源： <http://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/HDP-GPL/3.x/updates/3.1.0.0/HDP-GPL-3.1.0.0-centos7-gpl.tar.gz> <http://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/HDP-UTILS-1.1.0.22/repos/HDP-UTILS-1.1.0.22-centos7.tar.gz> <http://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/ambari/2.x/updates/2.7.3.0/ambari-2.7.3.0-centos7.tar.gz> <http://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/HDP/3.x/updates/3.1.0.0/HDP-3.1.0.0-centos7-rpm.tar.gz> <https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u222-b10/OpenJDK8U-jdk_x64_linux_hotspot_8u222b10.tar.gz>
# 1.软件使用的技术栈(编程语言列表、运行时、打包方法<可选>)
## 编程语言列表
C、C++、Java、HTML、Javascript、Shell
## 运行时
Oracle JDK1.8
## 编译环境
On Linux, you need the tools to create the native libraries: LZO headers,zlib headers, gcc, OpenSSL headers, cmake, protobuf dev tools, and libtool, and the GNU autotools (automake, autoconf, etc). For RHEL (and hence also CentOS): yum -y install lzo-devel zlib-devel gcc gcc-c++ autoconf automake libtool openssl-devel fuse-devel cmake For Debian and Ubuntu: apt-get -y install maven build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev libfuse-dev Windows： CygWin、jdk1.6+，Maven3.0+、Cmake 2.6+、Windows SDK or Visual Studio 2010 Professional、Cygwin、zlib headers 更全面的编译环境指导： <https://svn.apache.org/viewvc/hadoop/common/branches/branch-2/BUILDING.txt?view=markup>
## 打包方法
使用Maven打包，但是需要非常高配置的硬件，但是官方并没有给出具体的最低配置
# 2.能否运行于 ARM64v8 架构下
Arm64v8下的编译参考：<https://blog.csdn.net/github_39577257/article/details/106211612> 华为已经释出相关编译后的景象和repo： <https://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/>
# 3.是否已有中文语言包，没有的话能否汉化?
可以进行汉化，但是HDP组件无需汉化，因为其主要为后台进程，
# 4.如何进行二次开发，需要使用哪些技术，需要哪些软硬件环境，能否打包成安装 包<可选>?
不建议进行二次开发，如果非要二次开发，需要熟悉C、C++、Java、HTML、Javascript、Shell、深入研究java虚拟机和Linux操作系统
# 5.确定统一管理⻚面的需求边界
此处不涉及
# 6.调研过程中需要记录过程文档
文档在此处
# 7.列出面临的挑战与各种资源需求
## 面临的挑战
对Arm64V8的交叉编译环境不熟悉，对Linux生态尤其是国产OS的现状不了解
## 资源需求
更高配置的编辑服务器（x86_64）和测试开发环境服务器（Arm64V8）
# 8.著作权问题及法律⻛险
Apache License v2.0，可以商用，但是需要Notice文件标明引用了HDP
# 9.由于硬件暂时不具备，确定工作重心优先定制化;按照任务内容细化调研任务拆 分(大数据、k8s、数据库等)
# 10.统一用户管理，了解 apache kerberos(可选)，kuboard 的组、⻆色和用户
# 11.分任务确定各组件移植的工作量和移植后环境的稳定性(含压测《侧重⻓时间而 非数据量》)
# 12.部署步骤
## 下载离线部署包
<https://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/> ## 安全前置准备，包括安装操作系统、关闭防火墙、同步服务器时钟等；
### 机器名
vi /etc/hostname hostnamectl set-hostname xxx vi /etc/hosts
### 关闭防火墙
systemctl stop firewalld systemctl disable firewalld
### 同步服务器时钟
### selinux
在所有节点执行setenforce 0 命令，此处使用批处理shell执行： setenforce 0 集群所有节点修改 /etc/selinux/config 文件的如下内容： SELINUX=disabled
### 禁用虚拟内存和内存大页面
vim /etc/rc.d/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
禁用虚拟内存和内存大页面 echo vm.swappiness = 10 >> /etc/sysctl.conf echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag
### ntp
All server: yum -y remove chrony yum -y install ntpd
master(如果有自己的ntp上游就填写，否则就用本机器): vim /etc/ntp.conf
server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10
其他节点： vim /etc/ntp.conf
server 172.31.13.10
### repo
master： yum -y install httpd systemctl enable httpd systemctl start httpd
将下载的tar.gz拷贝到 /var/www/html 目录，然后解压缩各文件，删除原文件　tar.gz。 cd /var/www/html createrepo .
所有节点：
vim /etc/yum.repos.d/ambari.repo
[ambarirepo] name = ambari_repo baseurl = <http://master/ambari/centos7/2.7.3.0-139> enable = true gpgcheck = false
vi /etc/yum.repos.d/hdp.repo [hdprepo] name = hdp_repo baseurl = <http://master/HDP/centos7/3.1.0.0-78> enable = true gpgcheck = false
vi /etc/yum.repos.d/hdp_gpl.repo [hdpgplrepo] name = hdp_gpl_repo baseurl = <http://master/HDP-GPL/centos7/3.1.0.0-78> enable = true gpgcheck = false
vi /etc/yum.repos.d/hdp_utls_repo.repo [hdputilsrepo] name = hdp_utls_repo baseurl = <http://master/HDP-UTILS/centos7/1.1.0.22> enable = true gpgcheck = false
## 外部数据库如MySQL安装
yum -y install mariadb mariadb-server mysql-connector-java
create database metastore default character set utf8; CREATE USER 'hive'@'%' IDENTIFIED BY '1qaz2wsx'; GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; FLUSH PRIVILEGES; create database ambari default character set utf8;   CREATE USER 'ambari'@'%' IDENTIFIED BY '1qaz2wsx'; CREATE USER 'ambari'@'localhost' IDENTIFIED BY '1qaz2wsx'; CREATE USER 'ambari'@'master.test.fnii' IDENTIFIED BY '1qaz2wsx'; CREATE USER 'ambari'@'master' IDENTIFIED BY '1qaz2wsx';
GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'localhost'; GRANT ALL PRIVILEGES ON ambari.*TO 'ambari'@'master.test.fnii'; GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'master'; GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%'; FLUSH PRIVILEGES;
## 安装Ambari
### 安装ambari
yum -y install ambari-server
### 建立Ambari与MySQL驱动的连接
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
ambari-server setup Customize user account for ambari-server daemon [y/n] (n)? n
自定义jdk，输入java——home： /usr/java/jdk1.8.0_361-aarch64 Enter advanced database configuration [y/n] (n)? y Enter choice (1): 3 输入数据库相关信息 Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
### 完毕后手动执行sql创建ambari相关的表
mysql -uambari -p -Dambari < /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
### 启动
ambari-server start netstat -lnpt | grep 8080 ## 安装HDP集群；
### centos7免密登录
ssh-keygen -t rsa ssh-copy-id -i .ssh/id_rsa.pub -p22 root@localhost
### 访问
<http://masterIP:8080>
如果出现“Cannot find a valid baseurl for repo: HDP-3.1-repo-1” 的错误，按照下面的方式解决：
Steps
1) go to /usr/lib/ambari-server/web/javascipts
cd /usr/lib/ambari-server/web/javascripts
2) take backup of app.js
cp app.js app.js_backup
3) edit the app.js
find out the line(39892) : onNetworkIssuesExist: function () {
Change the line from :
/**
* Use Local Repo if some network issues exist */ onNetworkIssuesExist: function () { if (this.get('networkIssuesExist')) { this.get('content.stacks').forEach(function (stack) { stack.setProperties({ usePublicRepo: false, useLocalRepo: true }); stack.cleanReposBaseUrls(); }); } }.observes('networkIssuesExist'),
to
/**
* Use Local Repo if some network issues exist */ onNetworkIssuesExist: function () { if (this.get('networkIssuesExist')) { this.get('content.stacks').forEach(function (stack) { if(stack.get('useLocalRepo') != true){ stack.setProperties({ usePublicRepo: false, useLocalRepo: true }); stack.cleanReposBaseUrls(); } }); } }.observes('networkIssuesExist'),
as per : <https://github.com/apache/ambari/pull/2743/files>
Later as you have already deployed the cluster we need to reset the cluster (Caution : this will erase all the configs you have created previously in Step6 and also the Hosts and services you have selected need to select again )
Command :
ambari-server reset
And hard reload the page and start the create cluster wizard again.
Incase you have already at Step 9 and cannot proceed with ambari-server reset (as it invovles lots of Configs being added again , the below steps are for you )
Preqrequesties : The cluster now is in Deployment step(step 9 ) and you have only retry button to press
steps
1) Stop ambari-server
2) login to Database
3) use the below command to list out all the contents in repo_definition table :
select * from repo_definition;
4) you can see the base_url will be empty for the all the Rows in the table
5) Correct the base_url for every rows and update it using the command :
update repo_definition set base_url='<YOUR BASE URL>' where id=<THE CORESPONDING ID>;
for ex :
update repo_definition set base_url='http://asnaik.example.com/localrepo/HDP-3.1' where id=9;
update repo_definition set base_url='http://master/HDP/centos7/3.1.0.0-78' where id=25; update repo_definition set base_url='http://master/HDP-UTILS/centos7/1.1.0.22' where id=26;
6) after correcting all the base_url columns in repo_definition table and also delete the empty repos created by ambari from location /etc/yum.repos.d
7) start ambari, Login to UI and press retry button, The Installation will work as smooth as it can be.
Hope this helps.