1、版本说明
软件&镜像 | 版本&镜像信息 | 说明 |
---|---|---|
Kubernetes | v1.23.7 | k8s服务器 |
Kuboard | v3.5.2.0 | k8s连接管理工具 |
Canal | v1.1.6 | 数据同步 |
Canal-deployer | canal/canal-server:latest | canal-deplyer镜像版本信息 |
Canal-adapter | funnyzak/canal-adapter:1.1.6 | canal-adapter镜像版本信息 |
注意:
1> canal deployer由于官方发布的1.1.6版本不兼容jdk1.8(具体原因可参看相关issue,https://github.com/alibaba/canal/issues/4358),后续官方发布了latest版本兼容jdk1.8,所以这里deployer的镜像版本采用latest版本(官方canal deployer镜像地址:https://hub.docker.com/r/canal/canal-server/tags);当然,官方1.1.6版本是兼容jdk11及以上版本的,如果不想采用latest版本,也可以重新采用jdk11及以上版本自己构建docker镜像即可(这里提供一个非官方的基于jdk11的canal deployer 1.1.6版本镜像,仅供参考:funnyzak/canal-server:1.1.6)
2> 关于canal-adapter镜像说明,由于官方并未提供canal-adapter镜像,这部分查看issue也没有具体说明,鉴于canal 1.1.6版本存在不兼容jdk1.8版本的问题,所以当前docker hub上发布的基于jdk1.8版本的canal-adapter 1.1.6版本镜像,使用时均会发生部署失败的问题,因此这里使用了基于jdk11部署的canal-adapter镜像,关于此镜像说明可参看如下说明:https://github.com/funnyzak/canal-docker
3> 关于Kuboard说明,仅是一个k8s连接管理工具,如果介意,当然也可以采用原生方式部署,这里不做要求
2、MySQL数据库说明
MySQL数据库连接 | MySQL数据库版本 | 说明 |
---|---|---|
192.168.91.131:3306 | 5.7.34 | 源库连接地址 |
192.168.91.135:3306 | 5.7.34 | 目标库链接地址 |
3、k8s部署yml文件说明
1> 当前yml文件为两个MySQL数据库之间的同步,如果是MySQL和ES或者其他数据库之间的数据同步,此文档仅供参考;如果仅同步某个库下的某几张表数据,可以参考官方文档(https://github.com/alibaba/canal/wiki/Sync-RDB),修改对应配置,这里不再多做说明
2> 此yml文件部署方式为tcp方式,如使用kafka等MQ方式,请注意修改对应配置,此文档不再进行说明
3> 当前方式部署为增量数据同步,如需全量同步,可参考官方文档:https://github.com/alibaba/canal/wiki/ClientAdapter#324-%E6%89%8B%E5%8A%A8etl
4> 当前部署数据同步方式采用镜像同步,即源库和目标库之间的schema一致,从而实现两个MySQL数据库下的整个数据库中所有的数据表同步;若是单库单表数据同步,相关配置可参考官方文档(https://github.com/alibaba/canal/wiki/Sync-RDB),这里不再赘述
5> Canal相关配置通过configMap方式挂载,关于挂载到对应服务器文件的方式,感兴趣的小伙伴可以自行配置,这里不再赘述
6> 使用Kuboard工具可以直接复制如下部署文件,通过yml方式创建;当然通过其他k8s连接工具创建也是可以的,例如:kubesphere等
7> yml文件部署时,可以参考如下部署顺序,以减少部署失败的次数
(1)请先部署canal deployer、canal adapter对应的configMap文件
(2)之后再部署Zookeeper服务
(3)然后部署canal deployer服务,确保deployer服务部署启动成功
(4)最后再部署canal adapter服务
8> 部署过程中如果遇到部署问题,可查看issue寻找解决方案,官方issue地址:https://github.com/alibaba/canal/issues
3.1、Canal deployer部署文件
--- kind: Deployment apiVersion: apps/v1 metadata: labels: app: canal-deployer name: canal-deployer namespace: canal-mysql spec: replicas: 1 minReadySeconds: 10 selector: matchLabels: app: canal-deployer template: metadata: name: canal-deployer labels: app: canal-deployer spec: volumes: - name: conf configMap: defaultMode: 493 name: canal-deployer-configmap containers: - name: canal-deployer image: canal/canal-server:latest imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /home/admin/canal-server/conf/canal.properties name: conf subPath: canal.properties - mountPath: /home/admin/canal-server/conf/example/instance.properties name: conf subPath: instance.properties - mountPath: /home/admin/app.sh name: conf subPath: app.sh ports: - containerPort: 11111 protocol: TCP - containerPort: 11112 protocol: TCP --- kind: Service apiVersion: v1 metadata: name: canal-deployer namespace: canal-mysql labels: app: canal-deployer spec: ports: - port: 11111 name: deployer - port: 11112 name: metrics clusterIP: None selector: app: canal-deployer --- apiVersion: v1 kind: ConfigMap metadata: name: canal-deployer-configmap namespace: canal-mysql data: app.sh: |- #!/bin/bash set -e source /etc/profile export JAVA_HOME=/usr/java/latest export PATH=$JAVA_HOME/bin:$PATH touch /tmp/start.log host=`hostname -i` # waitterm # wait TERM/INT signal. # see: http://veithen.github.io/2014/11/16/sigterm-propagation.html waitterm() { local PID # any process to block tail -f /dev/null & PID="$!" # setup trap, could do nothing, or just kill the blocker trap "kill -TERM ${PID}" TERM INT # wait for signal, ignore wait exit code wait "${PID}" || true # clear trap trap - TERM INT # wait blocker, ignore blocker exit code wait "${PID}" 2>/dev/null || true } # waittermpid "${PIDFILE}". # monitor process by pidfile && wait TERM/INT signal. # if the process disappeared, return 1, means exit with ERROR. # if TERM or INT signal received, return 0, means OK to exit. waittermpid() { local PIDFILE PID do_run error PIDFILE="${1?}" do_run=true error=0 trap "do_run=false" TERM INT while "${do_run}" ; do PID="$(cat "${PIDFILE}")" if ! ps -p "${PID}" >/dev/null 2>&1 ; then do_run=false error=1 else sleep 1 fi done trap - TERM INT return "${error}" } function checkStart() { local name=$1 local cmd=$2 local timeout=$3 cost=5 while [ $timeout -gt 0 ]; do ST=`eval $cmd` if [ "$ST" == "0" ]; then sleep 1 let timeout=timeout-1 let cost=cost+1 elif [ "$ST" == "" ]; then sleep 1 let timeout=timeout-1 let cost=cost+1 else break fi done echo "start $name successful" } function start_canal() { echo "start canal ..." managerAddress=`perl -le 'print $ENV{"canal.admin.manager"}'` if [ ! -z "$managerAddress" ] ; then # canal_local.properties mode adminPort=`perl -le 'print $ENV{"canal.admin.port"}'` if [ -z "$adminPort" ] ; then adminPort=11110 fi su root -c 'cd /home/admin/canal-server/bin/ && sh restart.sh local 1>>/tmp/start.log 2>&1' sleep 5 #check start checkStart "canal" "nc 127.0.0.1 $adminPort -w 1 -z | wc -l" 30 else metricsPort=`perl -le 'print $ENV{"canal.metrics.pull.port"}'` if [ -z "$metricsPort" ] ; then metricsPort=11112 fi destination=`perl -le 'print $ENV{"canal.destinations"}'` if [[ "$destination" =~ ',' ]]; then echo "multi destination:$destination is not support" exit 1; else if [ "$destination" != "" ] && [ "$destination" != "example" ] ; then if [ -d /home/admin/canal-server/conf/example ]; then mv /home/admin/canal-server/conf/example /home/admin/canal-server/conf/$destination fi fi fi su root -c 'cd /home/admin/canal-server/bin/ && sh restart.sh 1>>/tmp/start.log 2>&1' sleep 5 #check start checkStart "canal" "nc 127.0.0.1 $metricsPort -w 1 -z | wc -l" 30 fi } function stop_canal() { echo "stop canal" su root -c 'cd /home/admin/canal-server/bin/ && sh stop.sh 1>>/tmp/start.log 2>&1' echo "stop canal successful ..." } function start_exporter() { su root -c 'cd /home/admin/node_exporter && ./node_exporter 1>>/tmp/start.log 2>&1 &' } function stop_exporter() { su root -c 'killall node_exporter' } echo "==> START ..." start_exporter start_canal echo "==> START SUCCESSFUL ..." tail -f /dev/null & # wait TERM signal waitterm echo "==> STOP" stop_canal stop_exporter echo "==> STOP SUCCESSFUL ..." canal.properties: >- canal.ip = canal.register.ip = canal.port = 11111 canal.metrics.pull.port = 11112 # 修改点: 此处值修改为部署的zookeeper服务的地址 canal.zkServers = zookeeper:2181 canal.withoutNetty = false canal.serverMode = tcp canal.destinations = example canal.auto.scan = true canal.instance.detecting.enable = false canal.instance.detecting.sql = select 1 canal.instance.detecting.interval.time = 3 canal.instance.detecting.retry.threshold = 3 canal.instance.detecting.heartbeatHaEnable = false canal.instance.transaction.size = 1024 canal.instance.fallbackIntervalInSeconds = 60 canal.instance.network.receiveBufferSize = 16384 canal.instance.network.sendBufferSize = 16384 canal.instance.network.soTimeout = 30 canal.instance.filter.druid.ddl = true canal.instance.filter.query.dcl = false canal.instance.filter.query.dml = false canal.instance.filter.query.ddl = false canal.instance.filter.table.error = false canal.instance.filter.rows = false canal.instance.filter.transaction.entry = false canal.instance.filter.dml.insert = false canal.instance.filter.dml.update = false canal.instance.filter.dml.delete = false canal.conf.dir = ../conf canal.instance.binlog.format = ROW,STATEMENT,MIXED canal.instance.binlog.image = FULL,MINIMAL,NOBLOB canal.instance.get.ddl.isolation = false canal.instance.parser.parallel = true canal.instance.parser.parallelThreadSize = 16 canal.instance.tsdb.enable = false canal.instance.tsdb.dir = ${canal.conf.dir}/tsdb canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL; canal.instance.tsdb.dbUsername = canal canal.instance.tsdb.dbPassword = canal canal.instance.tsdb.snapshot.interval = 24 canal.instance.tsdb.snapshot.expire = 360 canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml canal.instance.global.mode = spring canal.instance.global.lazy = false canal.instance.global.spring.xml = classpath:spring/default-instance.xml instance.properties: |- canal.instance.tsdb.enable = true canal.instance.gtidon = false # 修改点:将此处值修改为源库的地址 canal.instance.master.address = 192.168.91.131:3306 canal.instance.master.journal.name = canal.instance.master.position = canal.instance.master.timestamp= canal.instance.master.gtid= # 修改点: 将此处的用户名、密码修改为源库对应的用户名、密码 canal.instance.dbUsername = root canal.instance.dbPassword = root@123456 canal.instance.connectionCharset = UTF-8 canal.instance.enableDruid = false canal.instance.filter.regex=.*\\..* # 过滤控制台打印与配置表无关的日志信息 canal.instance.filter.black.regex=mysql\\.slave_.* canal.mq.topic = example canal.mq.partition = 0
说明:
1> configMap中的app.sh文件,主要是覆写了官方镜像中的app.sh文件,原因是因为我这里系统的用户名为 root,而官方镜像中的用户名为 admin,如果不覆写app.sh文件,会导致部署失败,具体失败原因可参看我的另一篇文章:https://www.cnblogs.com/cndarren/p/16746300.html
2> canal deployer挂载配置的路径与镜像中对应文件夹路径保持一致,镜像文件夹路径可参看:https://hub.docker.com/layers/canal/canal-server/latest/images/sha256-0d1018759efd92ad331c7cc379afa766c8d943ef48ef8d208ade646f54bf1565?context=explore
3.2、Canal adapter部署文件
--- kind: Deployment apiVersion: apps/v1 metadata: labels: app: canal-adapter name: canal-adapter namespace: canal-mysql spec: replicas: 1 minReadySeconds: 10 selector: matchLabels: app: canal-adapter template: metadata: name: canal-adapter labels: app: canal-adapter spec: volumes: - name: conf defaultMode: 420 configMap: name: canal-adapter-configmap containers: - name: canal-adapter image: funnyzak/canal-adapter:1.1.6 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /opt/canal/canal-adapter/conf/application.yml name: conf subPath: application.yml - mountPath: /opt/canal/canal-adapter/conf/bootstrap.yml name: conf subPath: bootstrap.yml - mountPath: /opt/canal/canal-adapter/conf/rdb/monitoralter.yml name: conf subPath: monitoralter.yml ports: - containerPort: 8081 protocol: TCP --- apiVersion: v1 kind: ConfigMap metadata: name: canal-adapter-configmap namespace: canal-mysql data: application.yml: |- server: port: 8081 logging: level: org.springframework: WARN com.alibaba.otter.canal.client.adapter.rdb: WARN spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null canal.conf: mode: tcp flatMessage: true # 修改点:zookeeper服务对应的地址 zookeeperHosts: zookeeper:2181 syncBatchSize: 1000 retries: 0 timeout: accessKey: secretKey: consumerProperties: # 修改点:由于采用tcp方式,所以这里配置就是tcp方式下canal deployer对应的地址 canal.tcp.server.host: canal-deployer:11111 canal.tcp.zookeeper.hosts: canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password: # 修改点:源库数据源配置信息 srcDataSources: monitorAlterDS: url: jdbc:mysql://192.168.91.131:3306/monitor_alert?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false username: root password: root@123456 # 修改点:目标库数据源配置信息 canalAdapters: - instance: example groups: - groupId: g1 outerAdapters: - name: logger - name: rdb key: mysql1 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://192.168.91.135:3306/monitor_alert?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false jdbc.username: root jdbc.password: admin@4321 bootstrap.yml: # 修改点:镜像方式同步配置文件 monitoralter.yml: |- dataSourceKey: monitorAlterDS destination: example groupId: g1 outerAdapterKey: mysql1 concurrent: true dbMapping: mirrorDb: true database: monitor_alert
说明:
1> canal adapter挂载配置的路径与镜像中对应文件夹路径保持一致,镜像文件夹路径可参看:https://hub.docker.com/layers/funnyzak/canal-adapter/1.1.6/images/sha256-1765db135c5e3610601f5680fa7d35f6b02f63218d8fdfe8b92d6371b1831190?context=explore
2> 如果有多个数据库下的数据表数据需要同步,可以在canal adapter的Deployment文件下添加对应的挂载的配置文件路径,然后再修改configMap文件即可,具体操作可参考如下所示
## 例如:需要再同步product数据库下的表数据 # Deployment文件修改如下: # 1、canal adapter的Deployment文件添加挂载的product数据库信息 containers: - name: canal-adapter image: funnyzak/canal-adapter:1.1.6 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /opt/canal/canal-adapter/conf/application.yml name: conf subPath: application.yml - mountPath: /opt/canal/canal-adapter/conf/bootstrap.yml name: conf subPath: bootstrap.yml - mountPath: /opt/canal/canal-adapter/conf/rdb/monitoralter.yml name: conf subPath: monitoralter.yml - mountPath: /opt/canal/canal-adapter/conf/rdb/product.yml name: conf subPath: product.yml # configMap修改如下: # 1、application.yml文件添加源库、目标库的地址信息 # 源库信息 srcDataSources: monitorAlterDS: url: jdbc:mysql://192.168.91.131:3306/monitor_alert?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false username: root password: root@123456 productDS: url: jdbc:mysql://192.168.91.131:3306/product?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false username: root password: root@123456 # 目标库信息 canalAdapters: - instance: example groups: - groupId: g1 outerAdapters: - name: logger - name: rdb key: mysql1 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://192.168.91.135:3306/monitor_alert?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false jdbc.username: root jdbc.password: admin@4321 - name: rdb key: mysql2 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://192.168.91.135:3306/product?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false jdbc.username: root jdbc.password: admin@4321 # 2、添加对应的挂载配置文件product.yml product.yml: |- dataSourceKey: productDS destination: example groupId: g1 outerAdapterKey: mysql2 concurrent: true dbMapping: mirrorDb: true database: product
3> 关于bootstrap.yml文件,这里并没有具体内容,原因仅是将canal adapter镜像中的bootstrap.yml内容进行注释,主要原因是canal adapter部署时并没有用到对应的canal_manager数据库配置,如果不覆写,会导致canal adapter部署启动时报连接不上canal_manager数据库的异常,具体问题可参看相关issue(https://github.com/alibaba/canal/issues/4197)
3.3、Zookeeper部署文件
--- kind: Deployment apiVersion: apps/v1 metadata: labels: app: zk-deployment name: zookeeper namespace: canal-mysql spec: replicas: 2 selector: matchLabels: app: zk template: metadata: name: zk labels: app: zk spec: volumes: - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai containers: - name: zookeeper image: zookeeper:3.6.2 imagePullPolicy: IfNotPresent volumeMounts: - name: localtime mountPath: /etc/localtime readOnly: true --- kind: Service apiVersion: v1 metadata: name: zookeeper namespace: canal-mysql labels: app: zk spec: ports: - port: 2181 name: client clusterIP: None selector: app: zk