首页 > 其他分享 >strimzi operator 部署kafka集群

strimzi operator 部署kafka集群

时间：2023-11-21 11:16:31浏览次数：36  
标签：strimzi kafka cluster io operator k8s

环境说明

本环境使用了单节点、临时存储集群的kafka-ephemeral-single配置。线上环境推荐kafka-persistent.yaml配置并修改storage配置为自动创建pv/pvc类型。

配置清单说明

1. kafka-ephemeral-single.yaml：非持久化存储，单节点集群；
2. kafka-ephemeral.yaml：非持久化存储，多节点集群；
3. kafka-jbod.yaml：jbod存储，多节点集群；
4. kafka-persistent-single.yaml：持久化存储，单节点集群；
5. kafka-persistent.yaml ：持久化存储，多节点集群；

创建名称空间

# kubectl create ns kafka

创建 strimzi-cluster-operator

创建 strimzi-cluster-operator 资源

# kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
configmap/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker created
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator created
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global created
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io created
serviceaccount/strimzi-cluster-operator created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client created
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation created
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkanodepools.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/strimzipodsets.core.strimzi.io created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation created
deployment.apps/strimzi-cluster-operator created

查看 strimzi-cluster-operator API资源

# kubectl api-resources |grep kafka
kafkabridges                      kb                                              kafka.strimzi.io/v1beta2                    true         KafkaBridge
kafkaconnectors                   kctr                                            kafka.strimzi.io/v1beta2                    true         KafkaConnector
kafkaconnects                     kc                                              kafka.strimzi.io/v1beta2                    true         KafkaConnect
kafkamirrormaker2s                kmm2                                            kafka.strimzi.io/v1beta2                    true         KafkaMirrorMaker2
kafkamirrormakers                 kmm                                             kafka.strimzi.io/v1beta2                    true         KafkaMirrorMaker
kafkanodepools                    knp                                             kafka.strimzi.io/v1beta2                    true         KafkaNodePool
kafkarebalances                   kr                                              kafka.strimzi.io/v1beta2                    true         KafkaRebalance
kafkas                            k                                               kafka.strimzi.io/v1beta2                    true         Kafka
kafkatopics                       kt                                              kafka.strimzi.io/v1beta2                    true         KafkaTopic
kafkausers                        ku                                              kafka.strimzi.io/v1beta2                    true         KafkaUser

查看 strimzi-cluster-operator pod

# kubectl get pods -n kafka
NAME                                        READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-5d6f48c6f9-s56hr   1/1     Running   0          5m1s

创建 kafka 集群

kafka-ephemeral-single.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
spec:
  kafka:
    version: 3.6.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.6"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

创建 kafka 集群

# kubectl apply -f kafka-ephemeral-single.yaml
kafka.kafka.strimzi.io/my-cluster created

查看 kafka pod

# kubectl get pods -n kafka
NAME                                          READY   STATUS    RESTARTS        AGE
my-cluster-entity-operator-75765b6dbb-l5s9q   3/3     Running   0               56s
my-cluster-kafka-0                            1/1     Running   0               81s
my-cluster-zookeeper-0                        1/1     Running   0               2m28s
my-cluster-zookeeper-1                        1/1     Running   0               2m28s
my-cluster-zookeeper-2                        1/1     Running   0               2m28s
strimzi-cluster-operator-5d6f48c6f9-vrfjp     1/1     Running   0               3m11s

查看 kafka svc

# kubectl get svc -n kafka
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
my-cluster-kafka-bootstrap    ClusterIP   10.100.139.196   <none>        9091/TCP,9092/TCP,9093/TCP                     16m
my-cluster-kafka-brokers      ClusterIP   None             <none>        9090/TCP,9091/TCP,8443/TCP,9092/TCP,9093/TCP   16m
my-cluster-zookeeper-client   ClusterIP   10.100.122.171   <none>        2181/TCP                                       17m
my-cluster-zookeeper-nodes    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP                     17m

测试 kafka 集群

发送消息

# kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic
If you don't see a command prompt, try pressing enter.
>hello
[2023-11-21 02:43:38,971] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {my-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>kafka
>

接收消息

# kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.
hello
kafka

删除 kafka 集群

# kubectl delete -f kafka-ephemeral-single.yaml
kafka.kafka.strimzi.io "my-cluster" deleted

删除 strimzi-cluster-operator

kubectl -n kafka delete -f 'https://strimzi.io/install/latest?namespace=kafka'
rolebinding.rbac.authorization.k8s.io "strimzi-cluster-operator" deleted
configmap "strimzi-cluster-operator" deleted
Warning: deleting cluster-scoped resources, not scoped to the provided namespace
clusterrole.rbac.authorization.k8s.io "strimzi-kafka-broker" deleted
customresourcedefinition.apiextensions.k8s.io "kafkaconnects.kafka.strimzi.io" deleted
clusterrole.rbac.authorization.k8s.io "strimzi-entity-operator" deleted
customresourcedefinition.apiextensions.k8s.io "kafkatopics.kafka.strimzi.io" deleted
clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-leader-election" deleted
customresourcedefinition.apiextensions.k8s.io "kafkamirrormaker2s.kafka.strimzi.io" deleted
rolebinding.rbac.authorization.k8s.io "strimzi-cluster-operator-entity-operator-delegation" deleted
clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-global" deleted
customresourcedefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" deleted
serviceaccount "strimzi-cluster-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "strimzi-cluster-operator" deleted
clusterrole.rbac.authorization.k8s.io "strimzi-kafka-client" deleted
customresourcedefinition.apiextensions.k8s.io "kafkausers.kafka.strimzi.io" deleted
clusterrolebinding.rbac.authorization.k8s.io "strimzi-cluster-operator-kafka-broker-delegation" deleted
customresourcedefinition.apiextensions.k8s.io "kafkaconnectors.kafka.strimzi.io" deleted
clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-watched" deleted
customresourcedefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" deleted
rolebinding.rbac.authorization.k8s.io "strimzi-cluster-operator-watched" deleted
customresourcedefinition.apiextensions.k8s.io "kafkarebalances.kafka.strimzi.io" deleted
rolebinding.rbac.authorization.k8s.io "strimzi-cluster-operator-leader-election" deleted
clusterrole.rbac.authorization.k8s.io "strimzi-cluster-operator-namespaced" deleted
customresourcedefinition.apiextensions.k8s.io "kafkamirrormakers.kafka.strimzi.io" deleted
customresourcedefinition.apiextensions.k8s.io "kafkanodepools.kafka.strimzi.io" deleted
customresourcedefinition.apiextensions.k8s.io "strimzipodsets.core.strimzi.io" deleted
clusterrolebinding.rbac.authorization.k8s.io "strimzi-cluster-operator-kafka-client-delegation" deleted
deployment.apps "strimzi-cluster-operator" deleted

参考文档

https://strimzi.io/quickstarts/

https://github.com/strimzi/strimzi-kafka-operator/tree/0.38.0/examples

标签：strimzi,kafka,cluster,io,operator,k8s
From： https://www.cnblogs.com/wangguishe/p/17845678.html

相关文章

  • Kafka异常——The coordinator is not available
    之前架设了一个Kafka集群，跑了很久没有什么错误，最近开发的小伙伴跟我说部分kafka不能消费了，了解详细情况后，自己也赶紧作了个测试，发现是有报错...Causedby:rg.apache.kafka.common.errors.CoordinatorNotAvailableException:Thecoordinatorisnotavailable....报错在网上......
  • kafka
    kafka下载路径:(https://kafka.apache.org/downloads)一、kafka单机安装1.1上传jdk环境jdk-8u202-linux-x64.tar.gzkafka_2.12-3.5.1.tgz1.2解压安装包tarxfjdk-8u202-linux-x64.tar.gz-C/usr/local/cd/usr/local/mvjdk1.8.0_202/java1.3编写环境变量文......
  • Canal+Kafka实现MySQL与Redis数据同步（二）
    Canal+Kafka实现MySQL与Redis数据同步（二）创建MQ消费者进行同步在application.yml配置文件加上kafka的配置信息：spring:kafka:#Kafka服务地址bootstrap-servers:127.0.0.1:9092consumer:#指定一个默认的组名group-id:consumer-group1......
  • kafka入门(一)：kafka消息消费
    安装kafka，创建topic：Windows安装kafka,详情见：https://blog.csdn.net/sinat_32502451/article/details/133067851Linux安装kafka，详情见：https://blog.csdn.net/sinat_32502451/article/details/133080353添加依赖包：<dependency><groupId>org.springfr......
  • kafka安装教程
    检查java8没有就安装java-version安装jdk1.8yum-yinstalljava-1.8.0-openjdk下载kafka（网速很慢）wgethttps://dlcdn.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz解压缩tar-xzfkafka_2.13-3.5.0.tgzcdkafka_2.13-3.5.0后台启动ZooKeeper服务（这里使用kafka里......
  • Introduing some cores concepts within kafka
    IntroductionToday,wetalkaboutsomecoresconceptswithinkafka.Apachekafkaisadistributedpublish-subscribemessagingsystem.ItisoriginallydevelopedatLinkdlnCorporationandlateronbecomeapartofApaccheproject,kafkaisafast,scalabl......
  • Kafka入门教程与详解（一）
    Kafka入门教程与详解（一）一、Kafka入门教程1.1消息队列（MessageQueue)MessageQueue消息传送系统提供传送服务。消息传送依赖于大量支持组件，这些组件负责处理连接服务、消息的路由和传送、持久性、安全性以及日志记录。消息服务器可以使用一个或多个代理实例。JMS（JavaMessaging......
  • Kafka 集群如何实现数据同步？
    哈喽大家好，我是咸鱼最近这段时间比较忙，将近一周没更新文章，再不更新我那为数不多的粉丝量就要库库往下掉了T﹏T刚好最近在学Kafka，于是决定写篇跟Kafka相关的文章（文中有不对的地方欢迎大家指出）考虑到有些小伙伴可能是第一次接触Kafka，所以先简单介绍一下什么是Kafka吧！Kafka......
  • Apache Kafka
    ApacheKafka®是 一个分布式流处理平台.这到底意味着什么呢?我们知道流处理平台有以下三种特性:可以让你发布和订阅流式的记录。这一方面与消息队列或者企业消息系统类似。可以储存流式的记录，并且有较好的容错性。可以在流式记录产生时就进行处理。Kafka适合什么样的场景?它可......
  • kafka第七天学习笔记
    在Kafka学习的第七天，你可能会进一步深入了解Kafka的特性和工作机制。以下是一些可能的学习点：Kafka的存储机制：Kafka使用一种称为“日志文件”的存储机制，将消息作为字节流存储在硬盘上。这种存储方式使得Kafka能够高效地处理大量的数据。消息的索引：Kafka为每个分区在硬盘上创建一个索......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99