Kafka在启用auto.create.topics时删除主题

Question

我们有一个两节点的kafka集群，具有auto。创造话题。启用和删除。话题启用设置为true。我们的应用程序读取公共请求主题，并响应客户端在请求负载中提供的响应主题。

auto.create.topics设置为true，因为我们的客户端具有自动缩放环境，其中新工作人员将从新响应主题中读取。由于客户端的一些实现问题，有很多创建的主题从未使用过（结束偏移量为0），我们正试图清理这些主题。

问题是，删除主题后，它几乎立即被重新创建。这些主题没有任何消费者（因为听它的工人已经死了）。

我已经尝试了以下方法

Kafka CLI delete命令

kafka-topics.sh --zookeeper localhost:2181 --topic <topic-name> --delete

在下创建zookeeper节点

/admin/delete_topics/<topic-name>

两者似乎都不起作用。在日志中，我看到收到了删除请求，并删除了相应的日志/索引。但在几秒钟/分钟内，主题将自动创建。日志供参考-

INFO [Partition <topic-name>-0 broker=0] No checkpointed highwatermark is found for partition <topic-name>-0 (kafka.cluster.Partition)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO [Partition <topic-name>-0 broker=0] <topic-name>-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-4 in /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-2 in /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-3 in /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-7 in /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-1 in /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-0 in /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-5 in /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-6 in /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete. (kafka.log.LogManager)
INFO [GroupCoordinator 0]: Removed 0 offsets associated with deleted partitions: <topic-name>-0. (kafka.coordinator.group.GroupCoordinator)
INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaFetcherManager)
INFO [ReplicaAlterLogDirsManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaAlterLogDirsManager)
INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaFetcherManager)
INFO [ReplicaAlterLogDirsManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaAlterLogDirsManager)
INFO Log for partition <topic-name>-0 is renamed to /home/ec2-user/data/kafka/0/<topic-name>-0.185c7eda12b749a2999cd39b3f90c738-delete and is scheduled for deletion (kafka.log.LogManager)
INFO Creating topic <topic-name> with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(0, 1)) (kafka.zk.AdminZkClient)
INFO [KafkaApi-0] Auto creation of topic <topic-name> with 1 partitions and replication factor 2 is successful (kafka.server.KafkaApis)
INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaFetcherManager)
INFO [Log partition=<topic-name>-0, dir=/home/ec2-user/data/kafka/0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
INFO [Log partition=<topic-name>-0, dir=/home/ec2-user/data/kafka/0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 0 ms (kafka.log.Log)
INFO Created log for partition <topic-name>-0 in /home/ec2-user/data/kafka/0 with properties {compression.type -> producer, message.format.version -> 2.2-IV1, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 86400000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
INFO [Partition <topic-name>-0 broker=0] No checkpointed highwatermark is found for partition <topic-name>-0 (kafka.cluster.Partition)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO [Partition <topic-name>-0 broker=0] <topic-name>-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)

当没有消费者在听，也没有制作人在制作主题时，有人知道重新创建主题背后的原因吗？复制可能是它背后的原因（可能是某些竞争条件）？我们正在使用Kafka2.2。

直接删除该主题的日志文件目录似乎可以工作，但是，当创建了数千个主题时，这很麻烦。我们希望有一个定期执行此操作的清理脚本，因为由于客户端环境的自动缩放特性，可能会有频繁的孤立响应主题。

使现代化

我尝试了Giorgos的建议，禁用了auto。创造话题。启用，然后删除主题。这一次该主题确实被删除了，但我的所有应用程序都没有出现任何错误（这使我得出结论，即该主题没有消费者/生产者）。

<罢工> 进一步，当 启用auto.create.topics.enable，并使用 复制因子=1，删除后不会重新创建主题。这让我相信，也许复制是罪魁祸首。这可能是Kafka的窃听器吗？ 在这里跳过枪；原来有些东西正在从客户环境中倾听/重新创建这些主题。

Answer 1

即使你提到没有消费者/生产者在消费/生产该主题，听起来情况也是如此。也许您在Kafka Connect上运行的连接器可以将数据从/复制到Kafka？

如果你仍然找不到是什么导致了删除的Kafka主题的重新创建，我建议将auto.create.topics.enable设置为false（暂时），这样主题就不能自动重新创建。然后导致主题重新创建的过程通常会失败，这将在日志中报告。