Spring Kafka批处理侦听器没有接收到超过1或2条消息

Question

Spring Kafka批处理消费者只能收到一两条消息，我们已经将fetch.min.bytes增加到9000

即使在增加值后，我们只收到1或2条消息，我们是否还需要增加fetch.min.bytes和fetch.max.wait.ms的值，或者我们是否需要添加任何其他配置，或者我们是否需要减少最大轮询记录大小？在本地环境中，我们收到10条消息，但在AWS MSK集群中，我们收到1或2条消息

消费者配置信息日志：

2022-05-13 16:15:48.118,"2022-05-13 16:15:48.117 [,] INFO main org.apache.kafka.clients.consumer.ConsumerConfig361  ConsumerConfig values: "
2022-05-13 16:15:48.118,    allow.auto.create.topics = true
2022-05-13 16:15:48.118,    auto.commit.interval.ms = 5000
2022-05-13 16:15:48.118,    auto.offset.reset = latest
2022-05-13 16:15:48.118,"   bootstrap.servers = [xyz.amazonaws.com:yyyy, xxxxxyzxx.amazonaws.com:yyyy, xxxxxxzzz.amazonaws.com:yyyy]"
2022-05-13 16:15:48.118,    check.crcs = true
2022-05-13 16:15:48.118,    client.dns.lookup = use_all_dns_ips
2022-05-13 16:15:48.118,    client.id = consumer-consumer.group.qa-5
2022-05-13 16:15:48.118,    client.rack = 
2022-05-13 16:15:48.118,    connections.max.idle.ms = 540000
2022-05-13 16:15:48.118,    default.api.timeout.ms = 60000
2022-05-13 16:15:48.118,    enable.auto.commit = false
2022-05-13 16:15:48.118,    exclude.internal.topics = true
2022-05-13 16:15:48.118,    fetch.max.bytes = 52428800
2022-05-13 16:15:48.118,    fetch.max.wait.ms = 5000
2022-05-13 16:15:48.118,    fetch.min.bytes = 9000
2022-05-13 16:15:48.118,    group.id = consumer.group.qa
2022-05-13 16:15:48.118,    group.instance.id = null
2022-05-13 16:15:48.118,    heartbeat.interval.ms = 3000
2022-05-13 16:15:48.118,    interceptor.classes = []
2022-05-13 16:15:48.118,    internal.leave.group.on.close = true
2022-05-13 16:15:48.118,    internal.throw.on.fetch.stable.offset.unsupported = false
2022-05-13 16:15:48.118,    isolation.level = read_uncommitted
2022-05-13 16:15:48.118,    key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
2022-05-13 16:15:48.118,    max.partition.fetch.bytes = 1048576
2022-05-13 16:15:48.118,    max.poll.interval.ms = 300000
2022-05-13 16:15:48.118,    max.poll.records = 200
2022-05-13 16:15:48.118,    metadata.max.age.ms = 300000

Answer 1

你能提供以下信息吗？

• MAX TPS（在主题中生成事件时）
• 每个事件的最大消息大小。

这些是理想的使用者配置值。

enable.auto.commit = false
auto.commit.interval.ms = 5000 <<Upto the Client, but this is ideal, if you enable Auto Commit>>
connections.max.idle.ms = 540000
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
heartbeat.interval.ms = 3000
max.poll.interval.ms = <<5 Sec is ideal>>
max.poll.records = <<Upto the Client>>
session.timeout.ms = 10000