CDH+Hive部署

CDH介绍：

   CDH 全称 Cloudera’s Distribution Including Apache Hadoop，是 Cloudera 公司提供的一个 Apache Hadoop 发行版。CDH 将 Hadoop 与其他十几个关键的开源项目集成，并且加入了集群自动化安装、中心化管理、集群监控、报警等功能。

Hive介绍：

   Hive是基于Hadoop的一个数据仓库工具，可以将结构化的数据文件映射为一张表，并提供类SQL查询功能。Hive处理的数据存储在HDFS，Hive分析数据底层的实现是MapReduce，执行程序运行在Yarn上。

优点

（1)操作接口采用类SQL语法，提供快速开发的能力（简单、容易上手）。
（2)避免了去写MapReduce，减少开发人员的学习成本。
（3)Hive的执行延迟比较高，因此Hive常用于数据分析，对实时性要求不高的场合。
（4)Hive优势在于处理大数据，对于处理小数据没有优势，因为Hive的执行延迟比较高
（5)Hive支持用户自定义函数，用户可以根据自己的需求来实现自己的函数。

缺点

• Hive的HQL表达能力有限

（1）迭代式算法无法表达
（2）数据挖掘方面不擅长，由于MapReduce数据处理流程的限制，效率更高的算法却无法实现。

• Hive的效率比较低

（1）Hive自动生成的MapReduce作业，通常情况下不够智能化
（2）Hive调优比较困难

节点规划：

   Cloudera Manager Server 节点提供 WebUI 操作界面，管理 Cloudera Manager Agent 节点，在运行 Agent 的机器上部署大数据相关组件。

版本信息：

   JDK：1.8.0

   Mysql：8.0.22

   Cloudera-Manager：6.3.1

   Hadoop：3.0.0

   Hive：2.1.1

   Hue：4.3.0

   Zookeeper：3.4.5

   Kafka：2.2.1

将所有所需程序包上传至服务器

环境准备：

1.设置主机名并添加到/etc/hosts文件内

hostnamectl set-hostname hadoop1
hostnamectl set-hostname hadoop2
hostnamectl set-hostname hadoop3

2.设置SSH免密登录

ssh-keygen -t rsa
然后敲（三个回车），就会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）*

将公钥拷贝到要免密登录的目标机器上

ssh-copy-id hadoop1
ssh-copy-id hadoop2
ssh-copy-id hadoop3

3.关闭防火墙

systemctl status firewalld

systemctl stop firewalld
systemctl enable firewalld

4.安装JDK(重要)

# 安装jdk
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 
# 添加环境变量
echo '# jdk' >> /etc/profile
echo 'export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera' >> /etc/profile
echo 'export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile
# 加载环境变量
source /etc/profile
java -version

5.安装MySQL

5.1查看MySQL是否安装

rpm -qa|grep mysql
# mysql-libs-5.1.73-7.el6.x86_64

5.2若有安装将其卸载

rpm -e --nodeps mysql-libs-5.1.73-7.el6.x86_64

删除原有MySql依赖(也适用安装出错干掉原来的) 
yum remove mysql-libs

5.3安装mysql初始化数据

5.3.1上传rpm包并安装（8.0或者5.7）

ls
# mysql-community-client-8.0.22-1.el7.x86_64.rpm
# mysql-community-client-plugins-8.0.22-1.el7.x86_64.rpm
# mysql-community-common-8.0.22-1.el7.x86_64.rpm
# mysql-community-libs-8.0.22-1.el7.x86_64.rpm
# mysql-community-libs-compat-8.0.22-1.el7.x86_64.rpm
# mysql-community-server-8.0.22-1.el7.x86_64.rpm

yum localinstall *.rpm -y

5.3.2修改配置文件/etc/my.cnf

[mysqld]
datadir=/var/lib/mysql/data/
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
#lower_case_table_names=1
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
#query_cache_limit = 8M
#query_cache_size = 64M
#query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/binlog/mysql-bin
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=11
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
prompt=mysql5729_db01 [\\d]>
sql_mode=STRICT_ALL_TABLES
socket=/var/lib/mysql/mysql.sock

5.3.3初始化mysql，初始化完成后注意查看密码

mysqld --defaults-file=/etc/my.cnf --initialize

# 设置忽略大小写,mysql8.0需要在初始化时设置（安装cdh不需要忽略）
mysqld --defaults-file=/etc/my.cnf --initialize --lower-case-table-names=1

5.3.4初始化后修改密码(以下方式选用一个)

set password='Root@123';
FLUSH PRIVILEGES;

# 或者
alter user 'root'@'%' identified with mysql_native_password by '密码';
UPDATE user SET Password=PASSWORD('root@123') where USER='root';

-- 8.0版本直接mysql -uroot -p连接
UPDATE user SET authentication_string=PASSWORD('root@123') where USER='root';
alter user 'boer'@'%'  IDENTIFIED BY 'Boer@123';  
alter user 'root'@'localhost' IDENTIFIED BY 'Root@123'; 
FLUSH PRIVILEGES;

更改权限(以下方式选用一个)

-- Mysql默认不允许远程登录，所以需要开启远程访问权限
update user set host = '%' where user = 'root';	
FLUSH PRIVILEGES;

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Root@123' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION; 
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION; 
FLUSH PRIVILEGES;

6.创建相关数据库

6.1mysql5.7

-- mysql5.7
-- scm
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Scm@147258';

-- amon
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Amon@147258';

-- rman
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Rman@147258';

-- hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Hue@147258';

-- hive
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'Hive@147258';

-- sentry
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;   
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Sentry@147258';

-- nav
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;      
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Nav@147258';

-- navms
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Navms@147258';

-- oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Oozie@147258';

-- flush
FLUSH PRIVILEGES;

6.2mysql8.0

-- mysql8.0
-- scm
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'scm'@'%' IDENTIFIED BY 'Scm@147258';
grant all privileges on scm.*  to 'scm'@'%' ;

-- amon
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'amon'@'%' IDENTIFIED BY 'Amon@147258';
grant all privileges on amon.*  to 'amon'@'%' ;

-- rman
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'rman'@'%' IDENTIFIED BY 'Rman@147258';
grant all privileges on rman.*  to 'rman'@'%' ;


-- hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
CREATE USER 'hue'@'%' IDENTIFIED BY 'Hue@147258';
grant all privileges on hue.*  to 'hue'@'%' ;


-- hive
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'hive'@'%' IDENTIFIED BY 'Hive@147258';
grant all privileges on metastore.*  to 'hive'@'%' ;


-- sentry
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'sentry'@'%' IDENTIFIED BY 'Sentry@147258';
grant all privileges on sentry.*  to 'sentry'@'%' ;   


-- nav
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
CREATE USER 'nav'@'%' IDENTIFIED BY 'Nav@147258';
grant all privileges on nav.*  to 'nav'@'%' ;     


-- navms
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'navms'@'%' IDENTIFIED BY 'Navms@147258';
grant all privileges on navms.*  to 'navms'@'%' ;


-- oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'Oozie@147258';
grant all privileges on oozie.*  to 'oozie'@'%' ;


-- flush
FLUSH PRIVILEGES;

7.上传连接mysql的jar包

ls
# mysql-connector-java-8.0.16.jar
修改其名称
cp mysql-connector-java-8.0.16.jar  /usr/share/java/mysql-connector-java.jar

8.安装 cloudera-manager

   创建cloudera-manager目录,存放cdh安装文件。解压压缩包

mkdir /opt/cloudera-manager
tar -zxvf cm6.3.1-redhat7.tar.gz
cd cm6.3.1/RPMS/x86_64/
mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/

注：rpm安装,所有节点都安装cloudera-manager-daemons和cloudera-manager-agent

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 

# 安装agent依赖
yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt

rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

8.1修改agent配置文件

vim /etc/cloudera-scm-agent/config.ini

server_host=hadoop1

8.2主节点安装cloudera-manager-server

rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm 

8.3修改 server的db.properties

vim /etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop1:3306
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=Scm@147258
com.cloudera.cmf.db.setupType=EXTERNAL

上传parcel包

ls /opt/cloudera/parcel-repo

# CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
# CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
# CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.torrent
# manifest.json

8.4初始化cm库

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h hadoop101 scm scm Scm@147258

9.启动Server与Agent

   启动主节点的Server和所有节点的Agent
启动server

service cloudera-scm-server.service start

启动Agent

systemctl start cloudera-scm-agent

启动Server查看日志，等待几分钟后无问题登录

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

10.登录：

默认端口号：7180
默认用户名：admin
默认密码：admin