Maxwell
Maxwell介绍
Maxwell:实时读取mysql的Binlog,生成json格式的消息,发送给kafka、redis等
下载地址:https://github.com/zendesk/maxwell/releases/download/v1.29.2/maxwell-1.29.2.tar.gz
再往后的版本废弃了jdk8
文档地址:https://github.com/zendesk/maxwell
Canal Maxwell对比
对比 | Canal | Maxwell |
---|---|---|
数据格式 | 自由 | json |
采集模式 | 增量 | 增量/全量 |
数据导出 | 定制 | 支持kafka等多种平台 |
HA | 支持 | 断点还原 |
量级 | 重量级 | 轻量级 |
多条数据 | LIST | 多条JSON |
Maxwell mysql配置
cd /opt/mysql/mysql/bin
./mysql -u root -p'123456'
create database maxwell;
CREATE USER maxwell IDENTIFIED BY 'maxwell';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'maxwell'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'maxwell'@'%' ;
FLUSH PRIVILEGES;
Maxwell 安装
mkdir /opt/maxwell/
cd /opt/maxwell/
wget https://github.com/zendesk/maxwell/releases/download/v1.29.2/maxwell-1.29.2.tar.gz
tar -xvf maxwell-1.29.2.tar.gz
mv maxwell-1.29.2 maxwell
mv maxwell-1.29.2.tar.gz /opt/pkg/
Maxwell输出到控制台
cd /opt/maxwell/maxwell/bin
##producer(stdout:控制台 kafka)
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --producer=stdout
Maxwell输出到kafka
cd /opt/maxwell/maxwell/
cp config.properties.example config.properties
vim config.properties
producer=kafka
kafka.bootstrap.servers=localhost:9092
host=localhost
user=maxwell
password=maxwell
cd /opt/maxwell/maxwell/bin
./maxwell --config ../config.properties
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --producer=kafka --kafka.bootstrap.server=127.0.0.1:9092 --kafka_topic=events
辅助命令
##kafka命令
cd /opt/kafka/kafka/bin/
./kafka-console-consumer.sh --topic events --from-beginning --bootstrap-server localhost:9092
delete from student;
insert into student values(1,'zhangsan',1);
insert into student values(2,'lisi',2);
insert into student values(3,'wangwu',3);
insert into student values(4,'sunliu',4);
insert into student values(5,'liuqi',5);
Maxwell kafka 多分区
cd /opt/maxwell/maxwell/
vim config.properties
kafka_topic=maxwell # [database, table, primary_key, transaction_id, thread_id, column]
producer_partition_by=table
##producer_partition_columns=id,foo,bar 如果上面是列需要开启
##producer_partition_by_fallback=database 如果上面是列需要开启
Maxwell 监控指定表
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --filter 'exclude:*.*,include:db.table' --producer=stdout
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --filter 'exclude:*.*,include:db.*' --producer=stdout
Maxwell 全量输出表
insert into maxwell.bootstrap(database_name,table_name)values('test','student');
--is_complete=0,就会同步出来所有的数据。输出完成后,mysql就会is_complete=1
结果数据:type:bootstrap-insert
标签:opt,producer,--,Maxwell,kafka,maxwell
From: https://www.cnblogs.com/wuxiaolong4/p/16805030.html