Kafka- S3接收器连接器-没有找到辅助反序列化器

Question

我试图使用来自kafka的消息，源消息以Avro格式序列化（我使用了AWS模式注册表）。

连接器配置：

{
        "name": "s3-sink-db01",
        "config": {
                "connector.class": "io.confluent.connect.s3.S3SinkConnector",
                "storage.class": "io.confluent.connect.s3.storage.S3Storage",
                "s3.bucket.name": "de-team",
                "name": "s3-sink-db01",
                "tasks.max": "3",
                "s3.region": "ap-south-1",
                "s3.part.size": "5242880",
                "s3.compression.type": "gzip",
                "timezone": "UTC",
                "locale": "en",
                "flush.size": "10",
                "rotate.interval.ms": "10",
                "topics.regex": "mysql-db01.(.*)",
                "internal.key.converter.schemas.enable": "false",
                "key.converter.schemas.enable": "false",
                "format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
                "internal.value.converter.schemas.enable": "false",
                "value.converter.schemas.enable": "false",
                "partitioner.class": "io.confluent.connect.storage.partitioner.HourlyPartitioner",
                "path.format": "YYYY/MM/dd/HH",
                "partition.duration.ms": "3600000",
 "key.converter": "com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter",
"value.converter": "com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter",
"key.converter.region": "ap-south-1",
"value.converter.region": "ap-south-1",
"key.converter.schemaAutoRegistrationEnabled": "true",
"value.converter.schemaAutoRegistrationEnabled": "true",
"key.converter.avroRecordType": "GENERIC_RECORD",
"value.converter.avroRecordType": "GENERIC_RECORD",
"internal.key.converter.schemas.enable": "false",
"key.converter.schemas.enable": "false",
"internal.value.converter.schemas.enable": "false",
"value.converter.schemas.enable": "false",
                "rotate.schedule.interval.ms": "3600000"
        }
}

但是当我尝试配置接收器连接器时，它会出现以下错误。

ERROR WorkerSinkTask{id=s3-sink-db01-2} Error converting message key in topic 'mysql-db01.devdb.table1' partition 0 at offset 0 and timestamp 1627302045505: Converting byte[] to Kafka Connect data failed due to serialization error:  (org.apache.kafka.connect.runtime.WorkerSinkTask:532)

 org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error:
...
...
...
 ERROR WorkerSinkTask{id=s3-sink-db01-2} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:184)

org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler

at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:206)

Caused by: org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error:

Caused by: com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Didn't find secondary deserializer.

Answer 1

这个问题是由于键/值转换器造成的。我已经通过纠正它解决了这个问题，一个依赖的问题在这里，解决方案在这里。