kafka默认参数情况下,以下数据,只有第一条和最后一条写kafka成功:
Will send data into kafka , filename :mysql_log_bin.000002, end position:722590822, send json size : 925442
Will send data into kafka , filename :mysql_log_bin.000002, end position:722695971, send json size : 1072672
Will send data into kafka , filename :mysql_log_bin.000002, end position:722801267, send json size : 1136702
Will send data into kafka , filename :mysql_log_bin.000002, end position:722907552, send json size : 1163697
Will send data into kafka , filename :mysql_log_bin.000002, end position:723012678, send json size : 1003731
Will send data into kafka , filename :mysql_log_bin.000002, end position:723116370, send json size : 936502
经过查询和测试,需要从kafka client,topic 和kafka server几方面分别配置解决限制问题:
1. kafka client(producer)配置:
max.request.size = 2097152 (2M)
buffer.memory = 33554432 (默认32M)
2. kafka server配置:
replica.fetch.max.bytes=20971520
message.max.bytes=10485760
max.message.bytes=10485760
可通过命令行查询配置成功:
$ bin/kafka-topics.sh --zookeeper localhost:2181 --topic 172.16.1.68 --describe
Topic:172.16.1.68 PartitionCount:1 ReplicationFactor:1 Configs:max.message.bytes=10485760