kafka
一、kafka单机安装
1.1 上传jdk环境
jdk-8u202-linux-x64.tar.gz
kafka_2.12-3.5.1.tgz
1.2 解压安装包
tar xf jdk-8u202-linux-x64.tar.gz -C /usr/local/
cd /usr/local/
mv jdk1.8.0_202/ java
1.3 编写环境变量文件
vim /etc/profile
#尾部添加
export JAVA_HOME=/usr/local/java
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
source /etc/profile
java -version
1.4 上传并解压
# 创建安装目录
mkdir /data
# 解压安装包
tar xf kafka_2.12-3.5.1.tgz -C /data/
mv /data/kafka_2.12-3.5.1/ /data/kafka
1.5 zookeeper配置
# 创建zookeeper目录
mkdir -p /data/kafka/zookeeper/data
# 编辑zookeeper配置文件
vim /data/kafka/config/zookeeper.properties
dataDir=/data/kafka/zookeeper/data
# 启动zookeeper
cd /data/kafka
./bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
# 验证zookeeper启动
netstat -utpln |grep 2181
1.6 kafka配置
# 创建kafka存放日志目录
mkdir -p /data/kafka/kafka.log
# 配置kafka
vim /data/kafka/config/server.properties
# 将文件的内容清空，替换为以下内容,注意IP、端口等其他配置
broker.id=0
listeners=PLAINTEXT://192.168.233.140:9092
port=9092
host.name=192.168.233.140
advertised.host.name=192.168.233.140
advertised.port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafka.log
num.partitions=5
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.233.140:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
# 启动kafka
cd /data/kafka
./bin/kafka-server-start.sh -daemon config/server.properties
#验证kafka启动
netstat -utpln |grep 9092
# 查看版本
[root@node1 bin]# ./kafka-server-start.sh --version
[2023-11-16 23:18:36,301] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
3.6.0
二、kafka集群安装
Kafka集群安装需要事先准备好JDK环境和Zookeeper集群环境,详见zookeeper集群安装配置
1. 下载安装包，解压
# 三节点一致
1. 下载
wget https://downloads.apache.org/kafka/3.6.0/kafka_2.12-3.6.0.tgz
2.解压
tar xf kafka_2.12-3.6.0.tgz -C /data
3. 改名
mv /data/kafka_2.12-3.6.0 /data/kafka
2. 修改配置
# 三节点
cd /data/kafka/conf
vim server.properties
broker.id=1 # 服务器id，唯一，三节点不同，分别设置为1，2，3
#listeners=PLAINTEXT://:9092 # 可根据需求修改，默认9092
host.name=192.168.233.128 # 添加，三节点不同，分别设置为各自IP
log.dirs=/data/kafka/kafka-logs # 数据存放目录
zookeeper.connect=192.168.233.128:2181,192.168.233.129:2181,192.168.233.130:2181 #配置zookeeper环境变量
3. 启动kafka
cd /data/kafka/bin
./kafka-server-start.sh -daemon ../config/server.properties
# 验证
netstat -utpln |grep 9092
4. zookeeper测试验证
cd /data/zookeeper/bin
./zkCli.sh -server 192.168.233.128:2181
[zk: 192.168.233.128:2181(CONNECTED) 4] ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]
[zk: 192.168.233.128:2181(CONNECTED) 5] ls /brokers
[ids, seqid, topics]
[zk: 192.168.233.128:2181(CONNECTED) 6] ls /brokers/ids
[1, 2, 3] # quit退出
5. 编写启动Kafka集群脚本
vim kfk_run_all.sh
#!/bin/bash
case $1 in
"start"){
for node in node1 node2 node3
do
echo "---------$node---------"
ssh $node "source /etc/profile;/data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties"
done
};;
"stop"){
for node in node1 node2 node3
do
echo "---------$node---------"
ssh $node "source /etc/profile;/data/kafka/bin/kafka-server-stop.sh"
done
};;
esac
6. 安装问题
启动报错：Error exists in the bytecode
解决方法：升级jdk版本