Kafka流再平衡:从再平衡到错误的状态转换

Question

我有4个单一分区和应用程序的三个实例的主题。我试图通过编写一个自定义的PartitionGrouper来实现可伸缩性，它将创建如下3个任务:

第一个实例-topic1，分区0，topic4，分区0

第二个实例-主题2，分区0

第三实例-桌面3，分区0

我将NUM_STANDBY_REPLICAS_CONFIG配置为1，因为它将在本地维护状态（也可以消除invalidstatestore异常）。

上述设置在两个实例中运行良好。当我把它增加到三个实例时，我开始面临重新平衡的问题。

StickyTaskAssignor:58 - Unable to assign 1 of 1 standby tasks for task [1009710637_0]. There is not enough available capacity. You should increase the number of threads and/or application instances to maintain the requested number of standby replicas.
    [INFO ] 2017-12-25 20:05:42.221 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] StreamThread:888 - stream-thread [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED.
    [INFO ] 2017-12-25 20:05:42.221 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] KafkaStreams:268 - stream-client [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449] State transition from REBALANCING to REBALANCING.
    [INFO ] 2017-12-25 20:05:42.276 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] StreamThread:195 - stream-thread [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] partition assignment took 55 ms.
    current active tasks: [1009710637_0]
    current standby tasks: [1240464215_0, 1833680710_0]
    previous active tasks: []
    [INFO ] 2017-12-25 20:05:42.631 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] StreamThread:939 - stream-thread [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] Shutting down
    [INFO ] 2017-12-25 20:05:42.631 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] StreamThread:888 - stream-thread [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] State transition from PARTITIONS_ASSIGNED to PENDING_SHUTDOWN.
    [INFO ] 2017-12-25 20:05:42.633 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] KafkaProducer:972 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
    [INFO ] 2017-12-25 20:05:42.638 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] StreamThread:972 - stream-thread [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] Stream thread shutdown complete
    [INFO ] 2017-12-25 20:05:42.638 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] StreamThread:888 - stream-thread [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] State transition from PENDING_SHUTDOWN to DEAD.
    [WARN ] 2017-12-25 20:05:42.638 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] KafkaStreams:343 - stream-client [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449] All stream threads have died. The Kafka Streams instance will be in an error state and should be closed.
    [INFO ] 2017-12-25 20:05:42.638 [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449-StreamThread-1] KafkaStreams:268 - stream-client [app-03-cfaf7841-dc19-4ee4-9d05-ae4928c21449] State transition from REBALANCING to ERROR.

Answer 1

我假设您的分区分组器破坏了某些东西。编写一个正确的自定义分区分组器非常困难，因为您需要了解很多关于Kafka Streams的内部结构。因此，首先不建议使用它。

错误本身意味着StandbyTask无法成功分配给线程，因为没有足够的线程。一般来说，这种思想是StandbyTask不能分配给运行相应“活动”任务的线程或同一stand by task的另一个副本:它不会增加容错能力，而只会浪费内存，就好像一个线程死了，所有任务都死了一样。

为什么会出现这个错误还不清楚(调试愉快:)。

但是，对于您的用例，您应该启动订阅单个主题的不同应用程序实例，并使用不同的application.id来横向扩展您的应用程序。