大数据平台及组件安装部署
实验一：Hadoop 全分布部署
-
实验任务一：Hadoop 集群验证
分布式集群搭建完成后，根据 Hadoop 两大核心组成，可以通过监测这 HDFS 分布式文件
系统和 MapReduce 来完成监测工作，通过以下步骤完成 Hadoop 集群测试：
（1）初始化集群，使用 Hadoop 命令启动集群。
（2）使用 Hadoop 命令，创建 HDFS 文件夹。
（3）使用 HDFS 命令查看文件系统“/”路径下是否存在文件。
（4）调用 Hadoop 自带的 WordCount 程序去测试 MapReduce，查看控制台是否能正确统
计单词数量。
1.1. 步骤一：具体测试内容
Hadoop 自身提供了许多功能用于监测，通过这些监控功能，可以判断出平台是否已经
完成全分布式集群搭建，以下步骤监测集群是否搭建成功：
（1）使用 JSP 查看各个节点启动的进程情况，都启动成功说明系统启动正常。
master（主节点）启动情况，命令如下：
[hadoop@master ~]$ jps
1649 DataNode
1506 NameNode
2309 NodeManager
2009 ResourceManager
2411 Jps
1838 SecondaryNameNode
slave1（从节点）节点启动情况，命令如下：
[hadoop@slave1 ~]# jps
2928 Jps
2640 NodeManager
2542 DataNode
slave2（从节点）节点启动情况，命令如下：
[hadoop@slave2 ~]# jps
2928 Jps
2640 NodeManager
2542 DataNode
（2）查看 Hadoop 的 Web 监控页面。
使用浏览器浏览主节点机http://master:50070,能查看NameNode节点状态说明系统启
动正常
使用浏览器浏览 master 节点 http://master:8088，查看所有应用说明系统启动正常，
结果
浏览 Nodes 说明系统启动正常，结果如
（3）使用 Hadoop 命令关闭集群。
使用命令关闭 Hadoop 集群，返回信息如下，说明系统关闭正常：
[hadoop@master ~]$ stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
slave2: stopping datanode
slave1: stopping datanode
192.168.10.20: ssh: connect to host 192.168.10.20 port 22: Connection refused
192.168.10.30: ssh: connect to host 192.168.10.30 port 22: Connection refused
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
slave1: stopping nodemanager
slave2: no nodemanager to stop
192.168.10.30: ssh: connect to host 192.168.10.30 port 22: Connection refused
192.168.10.20: ssh: connect to host 192.168.10.20 port 22: Connection refused
no proxyserver to stop
[hadoop@master ~]$ jps
2463 Jps
-
实验二：Sqoop 组件部署
-
实验任务一：Sqoop 数据传输验证
-
步骤一：查看 Sqoop 版本
通过 Sqoop 相关命令能够查询到 Sqoop 版本号为 1.4.7，则表示 Sqoop 部署成功，部署
成功信息如下：
[hadoop@master ~]$ sqoop version
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/05/11 03:08:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
-
步骤二：Sqoop 连接 MySQL 数据库
Sqoop 需要启动 Hadoop 集群，通过 Sqoop 连接 MySQL 查看数据库列表判断是否安装成
功，输入命令查看控制台最后输出是否为 MySQL 中的数据库。
[hadoop@master ~]$ sqoop list-databases --connect jdbc:mysql://127.0.0.1:3306/ --username root -P
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/04/11 23:27:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Enter password: # 此处需要输入 mysql 数据库的密码(我的密码为:Passwd123$)
23/04/11 23:27:32 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Tue Apr 11 23:27:32 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive
mysql
performance_schema
sys
-
步骤三：Sqoop 将 HDFS 数据导入到 MySQL
[hadoop@master ~]$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use sample ;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from student;
+--------+----------+
| number | name |
+--------+----------+
| 1 | zhanglin |
| 2 | lisi |
| 3 | yingliu |
| 4 | shengsan |
| 6 | ss2 |
| 8 | ss8 |
| 9 | ss9 |
+--------+----------+
7 rows in set (0.00 sec)
mysql> delete from student;
Query OK, 7 rows affected (0.01 sec)
mysql> select * from student;
Empty set (0.00 sec)
mysql> exit
Bye
#将 HDFS 数据导入到 MySQL
[hadoop@master ~]$ sqoop export --connect "jdbc:mysql://master:3306/sample?useUnicode=true&characterEncoding=utf-8" -username root --password Password123$ --table student --input-fieldsterminated-by ',' --export-dir /user/test Warning: /opt/sofeware/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/sofeware/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /opt/sofeware/sqoop/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 23/05/11 1454:10 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
23/05/11 14:55:01 INFO mapreduce.Job: Running job: job_1553997792849_0005
23/05/11 14:54:12 INFO mapreduce.Job: Job job_1553997792849_0005 running in uber mode : false
23/05/11 14:54:12 INFO mapreduce.Job: map 0% reduce 0%
23/05/11 14:54:25 INFO mapreduce.Job: map 100% reduce 0%
23/05/11 14:54:35 INFO mapreduce.Job: Task Id : attempt_1553997792849_0005_m_000000_0, Status : FAILED AttemptID:attempt_1553997792849_0005_m_000000_0 Timed out after 600 secs 23/05/11 1454: INFO mapreduce.Job: map 0% reduce 0%
23/05/11 1454: INFO mapreduce.Job: map 100% reduce 0%
（2）通过 MySQL 命令,查看数据是否成功导入 MySQL
[hadoop@master ~]$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use sample ;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
mysql> select * from student;
+--------+----------+
| number | name |
+--------+----------+
| 01 | zhanglin |
| 02 | lisi |
| 03 | yingliu |
| 04 | shengsan |
+--------+----------+
4 rows in set (0.00 sec)
mysql> quit;
Bye
-
实验二：Hive 组件部署
-
实验任务一：Hive 组件验证
-
步骤一：初始化 Hive
启动 Hadoop 集群的环境下，使用 Hive 初始化命令查看初始化控制台日志信息。
[hadoop@master ~]$ schematool -initSchema -dbType mysql
which: no hbase in (/usr/local/src/hive/bin:/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&us
eSSL=false
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Thu Mar 23 00:16:55 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Starting metastore schema initialization to 2.0.0
Initialization script hive-schema-2.0.0.mysql.sql
Thu Mar 23 00:16:56 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Initialization script completed
Thu Mar 23 00:17:07 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
schemaTool completed
#启动 hive
[hadoop@master ~]$ hive
which: no hbase in (/usr/local/src/hive/bin:/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
Thu Mar 23 00:17:49 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Mar 23 00:17:50 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Mar 23 00:17:50 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Mar 23 00:17:50 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Mar 23 00:17:53 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Mar 23 00:17:53 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Mar 23 00:17:53 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Thu Mar 23 00:17:53 EDT 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
OK
default
Time taken: 0.656 seconds, Fetched: 1 row(s)
hive> exit;