Kafka管理消费者群体之间的抵消

Question

假设我在一个2Kafka分区中有一个主题(test)，并且一个2consumer-group(X，Y)，每个consumer-group中只有一个consumer-group是消费主题。

现在，我想知道同一分区中其他使用者组的偏移量。下面的伪代码将解释需要

*** Let's assume this is running in the context of consumer group X

TOPIC = “test”
// consumer for group x
Consumer<K, V> consumerX = new KafkaConsumer<>(consumerProperties);
consumerX.subscribe(TOPIC, new ReportOnRebalance(……..));

// Get the current assigned partition, could be null but keep searching 
// until partition got assigned to the consumerX
Set<TopicPartition> topicPartition = consumerX.assignment();

// Get the last committed offset
offsetAndMetadataX = consumerX.committed(topicPartition)

// consumer for group y
Consumer<K, V> consumerY = new KafkaConsumer<>(consumerProperties);

// manually assign because I am interested in the offset for the 
// partition consumerX is going to serve for
consumerY.assign(topicPartition)

// Get the last committed offset
offsetAndMetadataY = consumerY.committed(topicPartition)

// Do require logic with offsetAndMetadataC and offsetAndMetadataY
newOffset = foo(offsetAndMetadataX, offsetAndMetadataY)

// want to reset the offset for this consumerY and in this 
// partition
consumerY.seek(topicPartition, bar(newOffset))

// Change offset for consumerX and starting polling for messages
consumerX.seek(topicPartition, newOffset)
while(...) {
    consumerX.poll(..)
    ....
}


*** Now the same code will run in the context of consumer group Y, but the role will be reversed

consumerY.subscribe()
consumerX.assign()
...
consumerY.seek(topicPartition, bar(newOffset))
...
// Change offset for consumerY and starting polling for messages
consumerY.seek(topicPartition, newOffset)
while(...) {
    consumerY.poll(..)
    ....
}

Answer 1

我没有尝试过您在这里描述的内容，但从官方文档来看，这似乎应该像您希望的那样工作：

https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/kafkaconsumer.html#ManualAssignment

此处突出显示的关键部分：

手动分区分配不使用组协调，因此使用者故障不会导致分配的分区重新平衡。每个使用者独立行动，即使它与另一个使用者共享一个groupId。为了避免偏移量提交冲突，您通常应该确保groupId对于每个使用者实例都是唯一的。

基本上，如果您开始手动为使用者分配分区，那么所有的动态重新平衡都会自动关闭。所以，你应该小心，但Kafka似乎确实考虑到了你描述的场景。