Kafka Connect 集群中只有一个节点响应 REST API 请求

Question

我在不同的主机上运行Kafka Connect集群，我只看到其中一个节点响应REST API请求（特别是：POST、PUT或DELETE请求）的行为。我可以通过关闭一个节点并向另一个活动节点发出写入命令来可靠地交换响应API请求的节点。

这是我的 docker-compose worker 配置：

version: '2'
services:
  connect:
    image: debezium/connect:1.1.0.Final
    ports:
     - 8083:8083
    volumes:
     - /etc/kafka/secrets:/etc/kafka/secrets
    environment:
     - BOOTSTRAP_SERVERS=my.region.aws.confluent.cloud:9092
     - GROUP_ID=debezium-postgres
     - CONFIG_STORAGE_TOPIC=dbz_pg_connect_configs
     - OFFSET_STORAGE_TOPIC=dbz_pg_connect_offsets
     - STATUS_STORAGE_TOPIC=dbz_pg_connect_statuses
     - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=3
     - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=3
     - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=3
     - OFFSET_FLUSH_INTERVAL_MS=8000
     - OFFSET_FLUSH_TIMEOUT_MS=60000
     - CONNECT_SECURITY_PROTOCOL=SASL_SSL
     - CONNECT_SASL_MECHANISM=PLAIN
     - CONNECT_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="<user>" password="<pass>";
     - CONNECT_PRODUCER_SECURITY_PROTOCOL=SASL_SSL
     - CONNECT_PRODUCER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="<user>" password="<pass>";
     - CONNECT_PRODUCER_SASL_MECHANISM=PLAIN
     - CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_SSL
     - CONNECT_CONSUMER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="<user>" password="<pass>";
     - CONNECT_CONSUMER_SASL_MECHANISM=PLAIN

我可以使用Debezium Postgres连接器和Kafka Snowflake连接器重现它。所以我认为问题在于Kafka Connect REST API本身，而不是任何特定的连接器库。

根据文件：

默认情况下，此服务在端口8083上运行。在分布式模式下执行时，REST API将是集群的主要接口。您可以向任何集群成员发出请求；如果需要，REST API会自动转发请求。

这是我的设置：

• 2个唯一的主机
• 2个运行Kafka Connect的文档化容器（图像实际上是debezium/连接：1.1.0。最终）
• 两者都在端口8083上运行REST服务。文档中没有迹象表明当容器在不同的主机上时这是一个问题

我看到的行为是：

• GET请求在任何情况下都可以同时对两个节点进行处理
• POST/PUT/DELETE请求在第一个节点上工作，以接受其中一个调用。在此之后，只有该节点响应POST/PUT/DELETE

另一个节点响应：

HTTP/1.1 100 Continue

HTTP/1.1 500 Internal Server Error
Date: Fri, 24 Apr 2020 17:59:55 GMT
Content-Type: application/json
Content-Length: 120
Server: Jetty(9.4.20.v20190813)

{"error_code":500,"message":"IO Error trying to forward REST request: java.net.SocketTimeoutException: Connect Timeout"}

编辑：这是Kafka Connect日志：

Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m 2020-04-24 19:11:14,934 ERROR  ||  IO error forwarding REST request:    [org.apache.kafka.connect.runtime.rest.RestClient]
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Connect Timeout
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:685)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:125)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:65)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:315)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:143)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:365)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:318)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.Server.handle(Server.java:494)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/java.lang.Thread.run(Thread.java:834)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m Caused by: java.net.SocketTimeoutException: Connect Timeout
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:802)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Apr 24 14:11:14 node01 docker-compose[57276]: #033[36mconnect_1  |#033[0m #011... 1 more

编辑：从我最初发布这个问题以来所学到的知识来看，很明显这不是一个错误或未记录的行为，而是与docker网络以及容器和独立主机之间的网络有关。但是，我仍然不清楚如何正确配置它，即使是一次性的。我们正在使用 nginx，并且在 5 个节点前面有一个 F5 负载均衡器。我能够从任一容器 ping 其他主机本身，因此主机至少可以相互通信。

Answer 1

它被记录在案...只有一个领导者将请求转发到底层配置/状态主题。类似于任何副本只有一个领导者主题分区。

除非您找到一个开放的 JIRA 或尝试配置与您的问题相关的每个属性，否则没有什么是错误。

特别是，您似乎没有设置< code > rest . advertised . listener (或公布的主机名)来允许每个服务器相互广播