Kafka连接JDBC源出记忆错误

Question

我正在使用 Confluent JDBC-Source 连接器运行以下作业：

{
    "name": "jobName",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
        "mode": "timestamp",
        "timestamp.column.name": "dateColumn",
        "topic.prefix": "connect-test.",
        "connection.password": "pw",
        "tasks.max": "1",
        "connection.user": "un",
        "poll.interval.ms": "300000",
        "name": "jobName",
        "connection.url": "jdbc:sqlserver://serverName;Database=dbName;user=un;password=pw",
        "table.whitelist": "tableName"
    }
}

我有一个类似的Kafka-Connect作业在同一个数据库和同一个用户上成功运行，但使用另一个较小的表。所以连接不是问题。

在运行作业的 Kafka-connect 服务器上的日志中，我看到以下内容：

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - connect-configs"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-coordinator-heartbeat-thread | connect-cluster"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - connect-offsets"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-3"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-4-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - connect-status"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DistributedHerder"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-5"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "org.eclipse.jetty.server.session.HashSessionManager@2c243a24Timer"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1434297727-21"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1434297727-26"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1434297727-27"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-14"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-15"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-4"

所以，没什么可说的。运行此作业的服务器现在无响应，并且不响应 REST 调用。有什么想法吗？

Answer 1

你需要

1. 通过SSH或以任何方式登录每个连接工作者
2. 强制停止Kafka Connect进程
3. 将KAFKA_HEAP_OPTS导出到比运行Kafka Connect的每台服务器上的默认值更大的东西。例如KAFKA_HEAP_OPTS="-Xms512M-Xmx4G"。假设Linux，那么最好的位置是kafka用户的bashrc文件。
4. 以您为
导出变量的用户身份再次运行 连接分布式

此外，您可能还希望导出更多值，以使JMX能够公开。这样，您可以在进程崩溃之前监视它