kafka学习之三_信创CPU下单节点kafka性能测试验证

背景

前面学习了 3controller+5broker 的集群部署模式.
晚上想着能够验证一下国产机器的性能. 但是国产机器上面的设备有限.
所以想着进行单节点的安装与测试. 并且记录一下简单结果

希望对以后的工作有指导意义

发现producer的性能比较与之前的 测试结果比较接近. 
但是consumer的测试结果看不出太多头绪来.
自己对kafka的学习还不够深入, 准备下一期就进行consumer的调优验证. 

测试结果验证

测试结果

广告一下自己的公众号

以申威为例进行安装说明

因为kafka 其实是基于java进行编写的消息队列.
所以不需要有繁杂的编译等过程. 
只要jdk支持, 理论上就可以进行运行. 
比较麻烦的就是稳定性和性能的表现. 
所以这里进行一下安装与验证. 

安装过程-1

上传文件到/root目录下面并且解压缩
cd /root && tar -zxvf kafka_2.13-3.5.0.tgz
然后编辑对应的文件:
cat > /root/kafka_2.13-3.5.0/config/kafka_server_jaas.conf <<EOF
KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="Testxxxxxx"
  user_admin="Testxxxxxx"
  user_comsumer="Testxxxxxx"
  user_producer="Testxxxxxx";
};
EOF

# 增加一个客户端配置文件 客户端才可以连接服务器端
cat > /root/kafka_2.13-3.5.0/config/sasl.conf <<EOF
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Testxxxxxx"; 
security.protocol=SASL_PLAINTEXT 
sasl.mechanism=PLAIN
EOF

安装过程-2

vim /root/kafka_2.13-3.5.0/config/kraft/server.properties
主要修改的点: 
因为是单节点所以比较简单了: 
process.roles=broker,controller
node.id=100
[email protected]:9094
listeners=SASL_PLAINTEXT://127.0.0.1:9093,CONTROLLER://127.0.0.1:9094
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.evervone.if.no.acl.found=true
advertised.listeners=SASL_PLAINTEXT://127.0.0.1:9093

# 最简单的方法可以删除配置文件的前面 50行, 直接放进去这些内容
vim->:1,50d->paste->edit_ips
其他ip地址可以如此替换:
sed -i 's/127.0.0.1/10.110.136.41/g'  /root/kafka_2.13-3.5.0/config/kraft/server.properties

安装过程-3

修改启动脚本:
vim /root/kafka_2.13-3.5.0/bin/kafka-server-start.sh 
在任意一个java opt 处增加:
-Djava.security.auth.login.config=/root/kafka_2.13-3.5.0/config/kafka_server_jaas.conf 

安装过程-4

# 初始化
/root/kafka_2.13-3.5.0/bin/kafka-storage.sh random-uuid
cd /root/kafka_2.13-3.5.0
bin/kafka-storage.sh format -t 7ONT3dn3RWWNCZyIwLrEqg  -c config/kraft/server.properties
# 启动服务
cd /root/kafka_2.13-3.5.0 && bin/kafka-server-start.sh -daemon  config/kraft/server.properties
# 创建topic
bin/kafka-topics.sh --create  --command-config config/sasl.conf  --replication-factor 1 --partitions 3 --topic zhaobsh01 --bootstrap-server 127.0.0.1:9093 
# 测试producer
bin/kafka-producer-perf-test.sh  --num-records 100000  --record-size 1024 --throughput -1 --producer.config config/sasl.conf  --topic zhaobsh01  --print-metrics --producer-props bootstrap.servers=127.0.0.1:9093
# 测试consumer
bin/kafka-consumer-perf-test.sh  --fetch-size 10000 --messages 1000000   --topic zhaobsh01 --consumer.config config/sasl.conf   --print-metrics --bootstrap-server 127.0.0.1:9093
# 查看日志
tail -f /root/kafka_2.13-3.5.0/logs/kafkaServer.out

申威的测试结果

producer:
100000 records sent, 7370.283019 records/sec (7.20 MB/sec), 2755.49 ms avg latency, 3794.00 ms max latency, 3189 ms 50th, 3688 ms 95th, 3758 ms 99th, 3785 ms 99.9th.
consumer:
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-06-20 21:56:02:493, 2023-06-20 21:56:39:755, 97.6563, 2.6208, 100000, 2683.6992, 5599, 31663, 3.0842, 3158.2604

飞腾的测试结果-hdd

producer:
100000 records sent, 1828.922582 records/sec (1.79 MB/sec), 6999.20 ms avg latency, 51476.00 ms max latency, 875 ms 50th, 21032 ms 95th, 21133 ms 99th, 21167 ms 99.9th.
consumer:

飞腾的测试结果-ssd

producer:
100000 records sent, 2219.706555 records/sec (2.17 MB/sec), 7073.51 ms avg latency, 41100.00 ms max latency, 1089 ms 50th, 20816 ms 95th, 20855 ms 99th, 20873 ms 99.9th.

海光的测试结果

producer:
100000 records sent, 40144.520273 records/sec (39.20 MB/sec), 486.67 ms avg latency, 681.00 ms max latency, 456 ms 50th, 657 ms 95th, 674 ms 99th, 678 ms 99.9th.
consumer:
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-06-20 22:28:04:364, 2023-06-20 22:28:21:274, 97.6563, 5.7751, 100000, 5913.6606, 3809, 13101, 7.4541, 7633.0051

鲲鹏的测试结果

producer:
100000 records sent, 42973.785991 records/sec (41.97 MB/sec), 463.69 ms avg latency, 621.00 ms max latency, 472 ms 50th, 593 ms 95th, 612 ms 99th, 619 ms 99.9th.
consumer:
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-06-20 22:33:58:168, 2023-06-20 22:34:15:595, 97.6563, 5.6037, 100000, 5738.2223, 3799, 13628, 7.1659, 7337.8339

倚天的测试结果

producer:
100000 records sent, 61162.079511 records/sec (59.73 MB/sec), 335.18 ms avg latency, 498.00 ms max latency, 326 ms 50th, 476 ms 95th, 494 ms 99th, 497 ms 99.9th.
consumer:
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-06-20 22:37:49:668, 2023-06-20 22:38:05:426, 97.6563, 6.1972, 100000, 6345.9830, 3597, 12161, 8.0303, 8223.008

AMD9T34的测试结果

producer:
100000 records sent, 74349.442379 records/sec (72.61 MB/sec), 253.07 ms avg latency, 364.00 ms max latency, 259 ms 50th, 344 ms 95th, 359 ms 99th, 363 ms 99.9th.
consumer:
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-06-20 22:44:14:446, 2023-06-20 22:44:29:058, 97.6563, 6.6833, 100000, 6843.6901, 3504, 11108, 8.7915, 9002.5207