使用者只从复制因子为3且分区为1的3代理配置中的特定代理读取

Question

我已经在一台机器上运行的容器集群中启动了多代理配置的所有组件。我使用了https://archive.apache.org/dist/kafka/2.0.0/kafka2.11-2.0.0.tgz中的shell脚本

1. 使用Zookeeper-properties启动zookeeper
2. 用3个不同的服务器属性启动了3个代理。它们仅在以下配置值上有所不同

broker.id
log.dirs
port

bin/kafka-topics.sh --create --topic repl_topic --zookeeper localhost:2181 --replication-factor 3 --partitions 1

bin/kafka-console-producer.sh --topic repl_topic --broker-list localhost:9092,localhost:9093,localhost:9094
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 -topic repl_topic --from-beginning

使用者只依赖于代理0。它不与其他人互动。

为什么？

Answer 1

我终于解决了这个问题，我已经解决了这个问题。检查__consumer_offsets，我注意到它没有被复制。

 bin/kafka-topics.sh --topic __consumer_offsets --zookeeper localhost:2181 --describe
Topic:__consumer_offsets    PartitionCount:50   ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets   Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 2    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 3    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 4    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 5    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 6    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 7    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 8    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 9    Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 10   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 11   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 12   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 13   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 14   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 15   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 16   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 17   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 18   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 19   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 20   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 21   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 22   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 23   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 24   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 25   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 26   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 27   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 28   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 29   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 30   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 31   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 32   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 33   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 34   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 35   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 36   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 37   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 38   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 39   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 40   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 41   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 42   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 43   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 44   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 45   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 46   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 47   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 48   Leader: 0   Replicas: 0 Isr: 0
    Topic: __consumer_offsets   Partition: 49   Leader: 0   Replicas: 0 Isr: 0

实际上，我第一次启动使用者是针对replication_factor等于1的主题。在那段时间里，消费者只创建了一个副本。该主题不再更新，然后如果Broker0关闭，其他broker将无法看到它。

可以通过命令bin/kafka-reassign-partitions.sh--zookeeper localhost:2181--reassignment-json-file new_reassignment.json--execute为每个分区建立新的复制级别

{"version":1,"partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":3,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":4,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":5,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":6,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":7,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":8,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":9,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":10,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":11,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":12,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":13,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":14,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":15,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":16,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":17,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":18,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":19,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":20,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":21,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":22,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":23,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":24,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":25,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":26,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":27,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":28,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":29,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":30,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":31,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":32,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":33,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":34,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":35,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":36,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":37,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":38,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":39,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":40,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":41,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":42,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":43,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":44,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":45,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":46,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":47,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":48,"replicas":[0,1,2],"log_dirs":["any","any","any"]},
{"topic":"__consumer_offsets","partition":49,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

本主题中的分区数默认设置为50。使用者触发其创建。主题主要由使用者用于提交每个主题的接收消息：Partion。如果这个__consumer_offsets主题对其他代理不可见，代理将无法通知使用者使用新消息。