使用Confluent Kafka Avro序列化程序的Spring Kafka测试找不到ZKClientConfig

Question

我很难开始嵌入式Kafka进行集成测试。将显示此错误：

@ExtendWith(SpringExtension.class)
@EmbeddedKafka(
        partitions = 1,
        bootstrapServersProperty = "spring.kafka.bootstrap-servers")
public class KafkaTest {

    @Test
    public void test(EmbeddedKafkaBroker broker) {
        var brokerList = broker.getBrokersAsString();
        log.debug("Embedded Kakfa topics {}", broker.getTopics());
    }
}

null

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.3.0.RELEASE</version>
    <relativePath/>
</parent>

当然，我在下面的pom.xml中列出了相关的依赖关系：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>5.2.3</version>
</dependency>

<!-- TEST SCOPE -->
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.6.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>org.junit.vintage</groupId>
            <artifactId>junit-vintage-engine</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <scope>test</scope>
</dependency>

我试图在org.springframework.kafka:spring-kafka-test和io.confluent:kafka-avro-serializer中排除org.apache.kafka:kafka-clients，但没有成功，出现了相同的错误。

<exclusion>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
</exclusion>

Answer 1

简短的回答：

这些依赖关系存在冲突：

• io.confluent:Kafka-Avro-Serializer:5.2.3
• org.springframework.kafka:spring-kafka-test:jar:2.5.0。release
• ...从父级org.springframework.boot:spring-boot-starter-parent:2.3.0.release

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>5.2.3</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
        </exclusion>
    </exclusions>
</dependency>

[INFO] +- io.confluent:kafka-avro-serializer:jar:5.2.3:compile  
[INFO] |  +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile    
[INFO] |  +- io.confluent:common-config:jar:5.2.3:compile   
[INFO] |  \- io.confluent:common-utils:jar:5.2.3:compile    
[INFO] |     +- org.apache.zookeeper:zookeeper:jar:3.4.13:compile   
[INFO] |     |  +- jline:jline:jar:0.9.94:compile   
[INFO] |     |  +- org.apache.yetus:audience-annotations:jar:0.5.0:compile  
[INFO] |     |  \- io.netty:netty:jar:3.10.6.Final:compile  
[INFO] |     \- com.101tec:zkclient:jar:0.10:compile

[INFO] \- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test    
[INFO]    +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test 
[INFO]    +- org.apache.kafka:kafka-streams:jar:2.5.0:test  
[INFO]    |  +- org.apache.kafka:connect-json:jar:2.5.0:test    
[INFO]    |  |  \- org.apache.kafka:connect-api:jar:2.5.0:test  
[INFO]    |  \- org.rocksdb:rocksdbjni:jar:5.18.3:test  
[INFO]    +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test   
[INFO]    +- org.apache.kafka:kafka_2.12:jar:2.5.0:test 
[INFO]    |  +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test  
[INFO]    |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test    
[INFO]    |  |     \- com.thoughtworks.paranamer:paranamer:jar:2.8:test 
[INFO]    |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test 
[INFO]    |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test   
[INFO]    |  +- com.yammer.metrics:metrics-core:jar:2.2.0:test  
[INFO]    |  +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test  
[INFO]    |  +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test   
[INFO]    |  +- org.scala-lang:scala-library:jar:2.12.10:test   
[INFO]    |  +- org.scala-lang:scala-reflect:jar:2.12.10:test   
[INFO]    |  +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test    
[INFO]    |  \- commons-cli:commons-cli:jar:1.4:test    
[INFO]    +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test    
[INFO]    \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test 
[INFO]       +- org.apiguardian:apiguardian-api:jar:1.1.0:test  
[INFO]       +- org.opentest4j:opentest4j:jar:1.2.0:test    
[INFO]       \- org.junit.platform:junit-platform-commons:jar:1.6.2:test

spring-kafka-test缺少org.apache.Zookeeper:Zookeeper，因此测试无法连接到Zookeeper，并显示错误：

由：org.springframework.beans.factory.BeanCreationException引起：创建名为“Embedded Kafka”的bean时出错：调用init方法失败；嵌套异常是java.lang.noClassDefFounderRorr：org/apache/zookeeper/client/zkClientConfig

从kafka-avro-serializer中排除这种依赖性可以解决这个问题（请参见简短的答案）。MVN dependency：tree的ouptut确认org.apache.zookeeper:zookeeper从kafka-avro-serializer中正确排除，并包含在spring-kafka-test中。

为Kafka-Avro-Serializer提取：

+- io.confluent:kafka-avro-serializer:jar:5.2.3:compile
[INFO] |  +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile
[INFO] |  +- io.confluent:common-config:jar:5.2.3:compile
[INFO] |  \- io.confluent:common-utils:jar:5.2.3:compile
[INFO] |     \- com.101tec:zkclient:jar:0.10:compile

为Spring-Kafka-test提取：

\- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test
[INFO]    +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test
[INFO]    +- org.apache.kafka:kafka-streams:jar:2.5.0:test
[INFO]    |  +- org.apache.kafka:connect-json:jar:2.5.0:test
[INFO]    |  |  \- org.apache.kafka:connect-api:jar:2.5.0:test
[INFO]    |  \- org.rocksdb:rocksdbjni:jar:5.18.3:test
[INFO]    +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test
[INFO]    +- org.apache.kafka:kafka_2.12:jar:2.5.0:test
[INFO]    |  +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test
[INFO]    |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test
[INFO]    |  |     \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO]    |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test
[INFO]    |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO]    |  +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO]    |  +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test
[INFO]    |  +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test
[INFO]    |  +- org.scala-lang:scala-library:jar:2.12.10:test
[INFO]    |  +- org.scala-lang:scala-reflect:jar:2.12.10:test
[INFO]    |  +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test
[INFO]    |  +- org.apache.zookeeper:zookeeper:jar:3.5.7:test
[INFO]    |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.5.7:test
[INFO]    |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:test
[INFO]    |  |  +- io.netty:netty-handler:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-common:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-resolver:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-buffer:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-transport:jar:4.1.49.Final:test
[INFO]    |  |  |  \- io.netty:netty-codec:jar:4.1.49.Final:test
[INFO]    |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.49.Final:test
[INFO]    |  |     \- io.netty:netty-transport-native-unix-common:jar:4.1.49.Final:test
[INFO]    |  \- commons-cli:commons-cli:jar:1.4:test
[INFO]    +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test
[INFO]    \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test
[INFO]       +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO]       +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO]       \- org.junit.platform:junit-platform-commons:jar:1.6.2:test