标签：opt producer -- Maxwell kafka maxwell
Maxwell
Maxwell介绍
Maxwell：实时读取mysql的Binlog，生成json格式的消息，发送给kafka、redis等
下载地址：https://github.com/zendesk/maxwell/releases/download/v1.29.2/maxwell-1.29.2.tar.gz
再往后的版本废弃了jdk8
文档地址：https://github.com/zendesk/maxwell
Canal Maxwell对比
| 对比 | Canal | Maxwell |
|---|
| 数据格式 |
自由 |
json |
| 采集模式 |
增量 |
增量/全量 |
| 数据导出 |
定制 |
支持kafka等多种平台 |
| HA |
支持 |
断点还原 |
| 量级 |
重量级 |
轻量级 |
| 多条数据 |
LIST |
多条JSON |
Maxwell mysql配置
cd /opt/mysql/mysql/bin
./mysql -u root -p'123456'
create database maxwell;
CREATE USER maxwell IDENTIFIED BY 'maxwell';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'maxwell'@'%';
GRANT ALL PRIVILEGES ON *.* TO 'maxwell'@'%' ;
FLUSH PRIVILEGES;
Maxwell 安装
mkdir /opt/maxwell/
cd /opt/maxwell/
wget https://github.com/zendesk/maxwell/releases/download/v1.29.2/maxwell-1.29.2.tar.gz
tar -xvf maxwell-1.29.2.tar.gz
mv maxwell-1.29.2 maxwell
mv maxwell-1.29.2.tar.gz /opt/pkg/
Maxwell输出到控制台
cd /opt/maxwell/maxwell/bin
##producer（stdout：控制台 kafka）
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --producer=stdout
Maxwell输出到kafka
cd /opt/maxwell/maxwell/
cp config.properties.example config.properties
vim config.properties
producer=kafka
kafka.bootstrap.servers=localhost:9092
host=localhost
user=maxwell
password=maxwell
cd /opt/maxwell/maxwell/bin
./maxwell --config ../config.properties
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --producer=kafka --kafka.bootstrap.server=127.0.0.1:9092 --kafka_topic=events
辅助命令
##kafka命令
cd /opt/kafka/kafka/bin/
./kafka-console-consumer.sh --topic events --from-beginning --bootstrap-server localhost:9092
delete from student;
insert into student values(1,'zhangsan',1);
insert into student values(2,'lisi',2);
insert into student values(3,'wangwu',3);
insert into student values(4,'sunliu',4);
insert into student values(5,'liuqi',5);
Maxwell kafka 多分区
cd /opt/maxwell/maxwell/
vim config.properties
kafka_topic=maxwell # [database, table, primary_key, transaction_id, thread_id, column]
producer_partition_by=table
##producer_partition_columns=id,foo,bar 如果上面是列需要开启
##producer_partition_by_fallback=database 如果上面是列需要开启
Maxwell 监控指定表
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --filter 'exclude:*.*,include:db.table' --producer=stdout
./maxwell --user='maxwell' --password='maxwell' --host='127.0.0.1' --filter 'exclude:*.*,include:db.*' --producer=stdout
Maxwell 全量输出表
insert into maxwell.bootstrap(database_name,table_name)values('test','student');
--is_complete=0，就会同步出来所有的数据。输出完成后，mysql就会is_complete=1
结果数据：type:bootstrap-insert
标签：opt,
producer,
--,
Maxwell,
kafka,
maxwell
From： https://www.cnblogs.com/wuxiaolong4/p/16805030.html