Kafka控制台使用者错误“分区上的偏移量提交失败”

Question

我正在使用kafka-console-consumer来探讨一个Kafka主题。

断断续续地，我会收到以下错误消息，然后是2个警告：

[2018-05-01 18:14:38,888] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-56648] Offset commit failed on partition my-topic-0 at offset 444: The coordinator is not aware of this member. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Asynchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=444, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

[2018-05-01 18:14:38,888] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56648] Synchronous auto-commit of offsets {my-topic-0=OffsetAndMetadata{offset=447, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

它在警告日志中建议：

这意味着对poll（）的后续调用之间的时间比配置的max.poll.interval.ms长，这通常意味着poll循环花费了太多时间处理消息。您可以通过增加会话超时，或者通过使用max.poll.records减小poll（）中返回的批的最大大小来解决这一问题。

因此，我需要增加max.poll.interval.ms或减少max.poll.records。

请告知每种方法的含义是什么，在不同的情况下哪一种更好？

Answer 1

如果您增加max.poll.interval.ms，说明“花时间处理大量记录是可以的”，并且如果您能够比较小的批处理更有效地处理较大的批处理，那么您将获得吞吐量。

为了减少max.poll.records表示“占用更少的记录，以便有足够的时间处理它们”，并且会倾向于延迟而不是吞吐量。

还要考虑这两个配置都很好，但在poll循环中有其他原因导致性能问题。在更改配置之前，我将首先探讨这一点，这样您就不会掩盖更大的问题。