03 Oct 2018
2018-07-09T13:54:25+02:00 INFO producer/broker/[[0]] maximum request accumulated, waiting for space 2018-07-09T13:54:25+02:00 INFO producer/broker/[[0]] maximum request accumulated, waiting for space
网上查看这个错误，找到一个解决办法的文章，里面提到解决办法是I have solved by increasing Buffer memory and producer config of Filebeat and Kafka。于是修改filebeat的配置
filebeat.prospectors: - type: log enabled: true paths: - /home/data/wwwlogs/my_web.log fields: service: my_web_nginx log_topic: filebeat-web-nginx multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:' multiline.negate: false multiline.match: after filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 name: 13.251.170.43 tags: ["web-nginx", "my"] output.kafka: hosts: ["*.*.*.138:9092", "*.*.*.138:9093", "*.*.*.74:9092"] topic: '%{[fields.log_topic]}' partition.round_robin: reachable_only: false required_acks: 1 compression: gzip max_message_bytes: 100000000
提高max_message_bytes的缓冲区数值为100000000。max_message_bytes配置说明