Kafka connect 2.0.0-无法在集群模式下创建连接器-错误：请求领导者重新配置连接器任务失败

Question

我已经在 2 个节点上部署了 Kafka connect （2.0.0） 集群，这是 Hortonworks 数据平台 3.1.4 的一部分。它被成功部署，kafka 连接也启动了。我能够使用 REST 命令通过 GET 方法列出连接器。

问题是，当我试图在集群模式下使用POST方法创建连接器时，它无法创建连接器，或者如果它确实创建了连接器，则无法创建任务。我在日志中看到以下错误

    [2020-02-14 15:45:41,622] ERROR [pool-3-thread-1] [org.apache.kafka.connect.runtime.distributed.DistributedHerder] - Request to leader to reconfigure connector tasks failed
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Failed to start RestClient: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:74)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:383)
        at org.eclipse.jetty.io.SelectorManager.execute(SelectorManager.java:143)
        at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:91)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:273)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:71)
        ... 6 more
[2020-02-14 15:45:41,622] ERROR [pool-3-thread-1] [org.apache.kafka.connect.runtime.distributed.DistributedHerder] - Failed to reconfigure connector's tasks, retrying after backoff:
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Failed to start RestClient: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:74)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.io.ManagedSelector@78075cca id=0 keys=0 selected=0
        at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:383)
        at org.eclipse.jetty.io.SelectorManager.execute(SelectorManager.java:143)
        at org.eclipse.jetty.io.ManagedSelector.doStart(ManagedSelector.java:91)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:273)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:71)
        ... 6 more

经过一些跟踪和错误，当任务在领导者连接节点上创建时，它似乎是有效的，如果它试图在非领导者连接节点上创建任务，那么它会抛出上述错误。我不太确定我的假设是否正确。

接下来，当我通过终止其他群集节点上的进程将连接群集强制进入独立连接节点时，创建的连接器和任务可以正常工作。

有人可以对这个问题提出一些看法，因为目前我无法在集群模式下使用 Kafka 连接。

感谢任何想法和投入。

Answer 1

经过一些跟踪和错误之后，似乎当在领导者连接节点上创建任务时，它会工作，如果它尝试在非领导者连接节点上创建任务，则会抛出上述错误。

不过，我不太确定我的假设是否正确。

它是。

您需要调整 Connect 工作线程的 rest.advertised.listener，以便他们了解彼此（通告网络中的 IP 或 DNS 名称）以及侦听器=HTTP：//0.0.0.0：8083

https://kafka.apache.org/documentation/#connectconfigs