首页 > 其他分享 >kubernetes 集群部署rabbimq3.11.11

kubernetes 集群部署rabbimq3.11.11

时间:2023-03-31 18:45:30浏览次数:67  
标签:11 rabbimq3.11 name kubernetes ## rabbitmq cluster internal rabbit

通过官方镜像 RabbitMQ Docker Image 和 rabbitmq-peer-discovery-k8s 插件进行集群部署。

0. 环境

 kubernetes 1.24

 rabbitmq3.11.11

1.命名空间

将 rabbitmq 的资源都放在 rabbitmq 命名空间内。

Namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: rabbitmq

2.配置

通过配置 configMap 将配置文件挂载到 rabbitmq 容器内 。

Config.yaml
 apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: rabbitmq
data:
  enabled_plugins: |
      [rabbitmq_management,rabbitmq_mqtt,rabbitmq_web_mqtt,rabbitmq_peer_discovery_k8s].
  rabbitmq.conf: |
      ## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more.
      cluster_formation.peer_discovery_backend  = k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      ## Service name is rabbitmq by default but can be overridden using the cluster_formation.k8s.service_name key if needed
      cluster_formation.k8s.service_name = rabbitmq-internal
      ## It is possible to append a suffix to peer hostnames returned by Kubernetes using cluster_formation.k8s.hostname_suffix
      cluster_formation.k8s.hostname_suffix = .rabbitmq-internal.rabbitmq.svc.cluster.local
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address?
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible.
      ## Set to "hostname" to use pod hostnames.
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME
      ## environment variable.
      cluster_formation.k8s.address_type = hostname
      ## How often should node cleanup checks run?
      cluster_formation.node_cleanup.interval = 30
      ## Set to false if automatic removal of unknown/absent nodes
      ## is desired. This can be dangerous, see
      ##  * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup
      ##  * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## See https://www.rabbitmq.com/ha.html#master-migration-data-locality
      queue_master_locator=min-masters
      ## This is just an example.
      ## This enables remote access for the default user with well known credentials.
      ## Consider deleting the default user and creating a separate user with a set of generated
      ## credentials instead.
      ## Learn more at https://www.rabbitmq.com/access-control.html#loopback-users
      loopback_users.guest = false
      ## https://www.rabbitmq.com/memory.html#configuring-threshold
      vm_memory_high_watermark.relative = 0.6


      ## On first start RabbitMQ will create a vhost and a user. These
      ## config items control what gets created.
      ## Relevant doc guide: https://rabbitmq.com/access-control.html
      ##
      default_vhost = /
      default_user = system
      default_pass = rbmqu0101081710



      # =======================================
      # MQTT section
      # =======================================

      ## TCP listener settings.
      ##
      # mqtt.listeners.tcp.1 = 127.0.0.1:61613
      # mqtt.listeners.tcp.2 = ::1:61613
      mqtt.listeners.tcp.default = 1883

      ## Set the default user name and password used for anonymous connections (when client
      ## provides no credentials). Anonymous connections are highly discouraged!
      ##
      mqtt.default_user = mqtt_admin
      mqtt.default_pass = rbmqmqtt_07231816

      ## Enable anonymous connections. If this is set to false, clients MUST provide
      ## credentials in order to connect. See also the mqtt.default_user/mqtt.default_pass
      ## keys. Anonymous connections are highly discouraged!
      ##
      mqtt.allow_anonymous = false

      ## If you have multiple vhosts, specify the one to which the
      ## adapter connects.
      ##
      mqtt.vhost = /

      ## Specify the exchange to which messages from MQTT clients are published.
      ##
      mqtt.exchange = exchange_mqtt_topic

      ## Specify TTL (time to live) to control the lifetime of non-clean sessions.
      ##
      mqtt.subscription_ttl = 1800000

      ## Set the prefetch count (governing the maximum number of unacknowledged
      ## messages that will be delivered).
      ##
      mqtt.prefetch = 10

这里配置了账号信息,cluster信息,mqtt信息(这个刚测试了一下,subscribe会失败)

3.密钥

通过 secrets 将 erlang-cookie 和默认用户信息写入到环境变量中。

Secret.yaml
 apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-secret
  namespace: rabbitmq
type: Opaque
data:
  RABBITMQ_ERLANG_COOKIE: MTIzajE5dWVkYXM3ZGFkODEwMjNqMTM5ZGph
  RABBITMQ_DEFAULT_USER: c3lzdGVt==
  RABBITMQ_DEFAULT_PASS: cmJtcXUwMTAxMDgxNzEw==

这里填写的数据是需要base64处理的,k8s会自动base64解开放到pod里面的环境变量

4.RBAC

rabbitmq-peer-discovery 需要 rabc 权限来获取 endpoints 信息来做集群节点的自动发现。

Rbac.yaml
 apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq
  namespace: rabbitmq
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq-peer-discovery-rbac
  namespace: rabbitmq
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq-peer-discovery-rbac
  namespace: rabbitmq
subjects:
- kind: ServiceAccount
  name: rabbitmq
  namespace: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rabbitmq-peer-discovery-rbac

5.服务

定义 headless service 作为 statefulset 的服务入口。

Service.yaml
 kind: Service
apiVersion: v1
metadata:
  namespace: rabbitmq
  name: rabbitmq-internal
  labels:
    app: rabbitmq
spec:
  clusterIP: None
  ports:
    - name: mqtt
      protocol: TCP
      port: 1883
    - name: epmd
      protocol: TCP
      port: 4369
    - name: amqp
      protocol: TCP
      port: 5672
    - name: amqp-tls
      protocol: TCP
      port: 5671
    - name: http
      protocol: TCP
      port: 15672
    - name: inter-node-cli
      protocol: TCP
      port: 25672
  selector:
    app: rabbitmq
---
kind: Service
apiVersion: v1
metadata:
  namespace: rabbitmq 
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  type: NodePort
  ports:
    - name: mqtt
      protocol: TCP
      port: 1883
      nodePort: 1883
    - name: amqp
      protocol: TCP
      port: 5672
      nodePort: 5672
    - name: http
      protocol: TCP
      port: 15672
      nodePort: 15672
  selector:
    app: rabbitmq 

6.持久卷

 statefulset 数据存储地方

PersistentVolume.yaml
 ---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbit-pv1
  labels:
    type: rabbitmq
spec:
  storageClassName: rabbitmq
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/opt/rabbitmq_data1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbit-pv2
  labels:
    type: rabbitmq
spec:
  storageClassName: rabbitmq
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/opt/rabbitmq_data2"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbit-pv3
  labels:
    type: rabbitmq
spec:
  storageClassName: rabbitmq
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/opt/rabbitmq_data3"

7. 有状态应用部署

按照官方集群部署的推荐方式使用 StatefulSet 方式部署,使用动态存储卷保存数据。

Statefulset.yaml
 apiVersion: apps/v1
# See the Prerequisites section of https://www.rabbitmq.com/cluster-formation.html#peer-discovery-k8s.
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: rabbitmq
spec:
  serviceName: rabbitmq-internal
  # Three nodes is the recommended minimum. Some features may require a majority of nodes
  # to be available.
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      nodeSelector:
        # Use Linux nodes in a mixed OS kubernetes cluster.
        # Learn more at https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#kubernetes-io-os
        kubernetes.io/os: linux
      initContainers:
        - name: fix-readonly-config
          image: busybox:1.31.1
          command:
            - sh
            - -c
            - cp /tmp/config/* /etc/rabbitmq;
          volumeMounts:
            - name: rabbitmq-config
              mountPath: /etc/rabbitmq
            - name: tmp-dir
              mountPath: /tmp/config
      containers:
        - name: rabbitmq
          image: rabbitmq:3.11.11
          # Learn more about what ports various protocols use
          # at https://www.rabbitmq.com/networking.html#ports
          ports:
            - name: mqtt
              protocol: TCP
              containerPort: 1883
            - name: epmd
              protocol: TCP
              containerPort: 4369
            - name: amqp
              protocol: TCP
              containerPort: 5672
            - name: amqp-tls
              protocol: TCP
              containerPort: 5671
            - name: http
              protocol: TCP
              containerPort: 15672
          livenessProbe:
            exec:
              # This is just an example. There is no "one true health check" but rather
              # several rabbitmq-diagnostics commands that can be combined to form increasingly comprehensive
              # and intrusive health checks.
              # Learn more at https://www.rabbitmq.com/monitoring.html#health-checks.
              #
              # Stage 2 check:
              command: ["rabbitmq-diagnostics", "status"]
            initialDelaySeconds: 60
            # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
            periodSeconds: 60
            timeoutSeconds: 15
          readinessProbe:
            exec:
              # This is just an example. There is no "one true health check" but rather
              # several rabbitmq-diagnostics commands that can be combined to form increasingly comprehensive
              # and intrusive health checks.
              # Learn more at https://www.rabbitmq.com/monitoring.html#health-checks.
              #
              # Stage 2 check:
              command: ["rabbitmq-diagnostics", "status"]
              # To use a stage 4 check:
              # command: ["rabbitmq-diagnostics", "check_port_connectivity"]
            initialDelaySeconds: 20
            periodSeconds: 60
            timeoutSeconds: 10
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: RABBITMQ_NODENAME
              value: rabbit@$(POD_NAME).rabbitmq-internal.$(POD_NAMESPACE).svc.cluster.local
            - name: RABBITMQ_USE_LONGNAME
              value: "true"
          envFrom:
            - secretRef:
                name: rabbitmq-secret
          volumeMounts:
            - name: rabbitmq-config
              mountPath: /etc/rabbitmq
            - name: rabbitmq-data
              mountPath: /var/lib/rabbitmq
      volumes:
        - name: rabbitmq-config
          emptyDir: {}
        - name: tmp-dir
          configMap:
            name: rabbitmq-config
  volumeClaimTemplates:
    - metadata:
        name: rabbitmq-data
        namespace: rabbitmq
        labels:
          app: rabbitmq
      spec:
        accessModes:
          - "ReadWriteOnce"
        storageClassName: rabbitmq
        resources:
          requests:
            storage: 10Gi

8. 部署

sudo kubectl create -f Namespace.yaml
sudo kubectl create -f PersistentVolume.yaml
sudo kubectl create -f Rbac.yaml
sudo kubectl create -f Secret.yaml
sudo kubectl create -f Config.yaml
sudo kubectl create -f Statefulset.yaml
sudo kubectl create -f Service.yaml

9. 查看pods

qiteck@server:~/program/rabbitmq/3.11.11/k8s$ sudo kubectl get pods -n rabbitmq -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
rabbitmq-0   1/1     Running   0          32m   10.244.1.133   server   <none>           <none>
rabbitmq-1   1/1     Running   0          31m   10.244.1.134   server   <none>           <none>
rabbitmq-2   1/1     Running   0          30m   10.244.1.135   server   <none>           <none>

10. 查看服务

qiteck@server:~/program/rabbitmq/3.11.11/k8s$ sudo kubectl get service -n rabbitmq -o wide
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                   AGE   SELECTOR
rabbitmq            NodePort    10.96.172.247   <none>        1883:1883/TCP,5672:5672/TCP,15672:15672/TCP               32m   app=rabbitmq
rabbitmq-internal   ClusterIP   None            <none>        1883/TCP,4369/TCP,5672/TCP,5671/TCP,15672/TCP,25672/TCP   32m   app=rabbitmq

11. 管理系统查看

12. 集群查看

12.1. rabbitmqctl cluster_status

    root@rabbitmq-0:/# rabbitmqctl cluster_status
    Cluster status of node [email protected] ...
    Basics
    
    Cluster name: [email protected]
    Total CPU cores available cluster-wide: 6
    
    Disk Nodes
    
    [email protected]
    [email protected]
    [email protected]
    
    Running Nodes
    
    [email protected]
    [email protected]
    [email protected]
    
    Versions
    
    [email protected]: RabbitMQ 3.11.11 on Erlang 25.3
    [email protected]: RabbitMQ 3.11.11 on Erlang 25.3
    [email protected]: RabbitMQ 3.11.11 on Erlang 25.3
    
    CPU Cores
    
    Node: [email protected], available CPU cores: 2
    Node: [email protected], available CPU cores: 2
    Node: [email protected], available CPU cores: 2
    
    Maintenance status
    
    Node: [email protected], status: not under maintenance
    Node: [email protected], status: not under maintenance
    Node: [email protected], status: not under maintenance

12.2. 启动日志查看

tail -f /var/log/rabbitmq/rabbit\@rabbitmq-0.rabbitmq-internal.rabbitmq.svc.cluster.local.log:
    
    Feature flags
    
    Flag: classic_mirrored_queue_version, state: enabled
    Flag: classic_queue_type_delivery_support, state: enabled
    Flag: direct_exchange_routing_v2, state: enabled
    Flag: drop_unroutable_metric, state: enabled
    Flag: empty_basic_get_metric, state: enabled
    Flag: feature_flags_v2, state: enabled
    Flag: implicit_default_bindings, state: enabled
    Flag: listener_records_in_ets, state: enabled
    Flag: maintenance_mode_status, state: enabled
    Flag: quorum_queue, state: enabled
    Flag: stream_queue, state: enabled
    Flag: stream_single_active_consumer, state: enabled
    Flag: tracking_records_in_ets, state: enabled
    Flag: user_limits, state: enabled
    Flag: virtual_host_metadata, state: enabled
    root@rabbitmq-0:/# tail -f /var/log/rabbitmq/rabbit\@rabbitmq-0.rabbitmq-internal.rabbitmq.svc.cluster.local.log
    2023-03-31 08:33:03.820636+00:00 [info] <0.724.0> Server startup complete; 5 plugins started.
    2023-03-31 08:33:03.820636+00:00 [info] <0.724.0>  * rabbitmq_peer_discovery_k8s
    2023-03-31 08:33:03.820636+00:00 [info] <0.724.0>  * rabbitmq_peer_discovery_common
    2023-03-31 08:33:03.820636+00:00 [info] <0.724.0>  * rabbitmq_management
    2023-03-31 08:33:03.820636+00:00 [info] <0.724.0>  * rabbitmq_web_dispatch
    2023-03-31 08:33:03.820636+00:00 [info] <0.724.0>  * rabbitmq_management_agent
    2023-03-31 08:34:03.957323+00:00 [info] <0.638.0> node '[email protected]' up
    2023-03-31 08:34:07.136896+00:00 [info] <0.638.0> rabbit on node '[email protected]' up
    2023-03-31 08:35:07.762592+00:00 [info] <0.638.0> node '[email protected]' up
    2023-03-31 08:35:10.314678+00:00 [info] <0.638.0> rabbit on node '[email protected]' up

13. 遇到问题

13.1. this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'

  服务还没有起来很多命令不可以执行,

  k8s的PostStart :这个钩子在容器创建后立即执行,这个时候rabbit还没有起来 

13.2. pvc异常:storageclass.storage.k8s.io "rabbitmq" not found: 访问模式,存储类型要对

  PersistentVolume的accessModes和storageClassName必须要和Statefulset里面的volumeClaimTemplates一致

 

13.3. Error: secret "rabbitmq-secret" not found: 

  secret不存在,创建一下就好

13.4.Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 12

  secret里面配置的数据需要base64处理

13.5.Feature flags: `maintenance_mode_status`: required feature flag not enabled!  It must be enabled before upgrading RabbitMQ.

  后面才发现是集群重新配置以后,之前的PersistentVolume没有删除导致,集群重启的话,需要彻底清空PersistentVolume的数据

13.6.Node '[email protected]'  thinks it's clustered with node '[email protected]',but '[email protected]' disagrees

  后面才发现是集群重新配置以后,之前的PersistentVolume没有删除导致,集群重启的话,需要彻底清空PersistentVolume的数据

标签:11,rabbimq3.11,name,kubernetes,##,rabbitmq,cluster,internal,rabbit
From: https://www.cnblogs.com/zhanchenjin/p/17277153.html

相关文章

  • windows使用/服务(10)Win11预览版退回正式版方法介绍
    闲的没事把系统win11:21H2正式稳定版本升级到22H2内部测试版,不太习惯而且还是个滚动发行版不稳定,回退到之前版本Win11将预览版返回到正式版的方法1.右击“计算机”选择“属性”,选择“winodws更新”选择“高级选项”。2.然后点击“恢复”,再点击“高级启动”下的“重启”。3.......
  • 在windows 11家庭版 系统上安装linux子系统
    步骤:1.启动WSL单击Win11任务栏上给出的“搜索图标”类型–打开或关闭窗口功能选择它,当“WindowsSubsystemforLinux”时。按确定按钮。然后“重启”你的系统.2.如果机器比较新,一般从WSL1升级到WSL2再进行操作。WS2可以通过命令行直接安装系统。可节省第三步。但有时升级......
  • kubernetes Secret使用
    节选rabbitmq的k8s部署部分secret用来配置环境变量1.Secret.yaml:1.1.配置文件secret.yaml apiVersion:v1kind:Secretmetadata:name:rabbitmq-secretnamespace:rabbitmqtype:Opaquedata:RABBITMQ_ERLANG_COOKIE:MTIzajE5dWVkYXM3ZGFkODEwMjNqMTM5ZGph......
  • 对于11g新特性自适应游标共享
    绑定变量优缺点、使用、绑定变量窥探、Oracle自适应共享游标_ITPUB博客自适应游标共享虽然解决了一些由于10g中绑定变量窥探只进行首次硬解析才窥探生成执行计划,由于数据倾斜造成执行计划不准确问题,但是不是每次执行sql都进行窥探,游标共享是“自适应”的,也会出现执行计划不准确......
  • 01011 IDEA创建新项目
    创建新项目​ new->project创建新模块​ projectstructure->+->newmodule->改名字不改路径模块下创建包new->package->com.lxc.->作用会生成对应文件夹包下创建类​ new->class->name......
  • 基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试 - 3/3
    基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试-1/3基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试-2/3基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试-3/3项目地址:https://github.com/janrs-io/Jgrpc转载请注明来源:https://janrs.com/6rdh......
  • 基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试 - 2/3
    基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试-1/3基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试-2/3基于Go/Grpc/kubernetes/Istio开发微服务的最佳实践尝试-3/3项目地址:https://github.com/janrs-io/Jgrpc转载请注明来源:https://janrs.com/ugj7......
  • 永磁同步直线电机仿真实例,仿真教学 maxwell16.0版本 12槽11极
    永磁同步直线电机仿真实例,仿真教学maxwell16.0版本12槽11极包括图中模型以及一个仿真设置要点word文档教程YID:99150643350855880......
  • 《百度移动互联网发展趋势报告2011 年Q3》概要
    在最新发布的《百度移动互联网发展趋势报告2011 年 Q3》中,百度无线和易观国际依托百度移动搜索数据,结合易观智库的应用数据补充,从用户分布及网络接入、移动终端竞争格局及趋势、移动终端平台变化趋势、手机浏览器竞争格局及发展趋势、应用商店现状及发展趋势五个方面,对当前国内......
  • HCIP-ICT实战进阶11-虚拟路由网关协议VRRP
    HCIP-ICT实战进阶11-虚拟路由网关协议VRRP0回顾链路聚合是为了做什么?通过多条线路实现冗余备份,可以实现N+M备份,同时链路聚合可以提升网络速度.二层环境通过各种生成树协议(多为MSTP)避免环路.在IGP协议中,三层环境可以通过自身算法实现冗余备份比如SPF算法.......