JAVAlang.NoClassDefFoundError:scala/collection/GenTraversableOnce$class

Question

我试图实现Apache kafka和火花流集成这里是我的python代码：

from __future__ import print_function
import sys
from pyspark.streaming import StreamingContext
from pyspark import SparkContext,SparkConf
from pyspark.streaming.kafka import KafkaUtils

if __name__ == "__main__":
#conf = SparkConf().setAppName("Kafka-Spark").setMaster("spark://127.0.0.1:7077")
conf = SparkConf().setAppName("Kafka-Spark")
#sc = SparkContext(appName="KafkaSpark")
sc = SparkContext(conf=conf)
stream=StreamingContext(sc,1)
map1={'demo':1}
kafkaStream = KafkaUtils.createStream(stream, 'localhost:2181', "test-consumer-group", map1)

# kafkaStream = KafkaUtils.createStream(stream, 'localhost:2181', "name", map1) #tried with localhost:2181 too
lines = kafkaStream.map(lambda x: x[1])
counts = lines.flatMap(lambda line: line.split(" ")) 
     .map(lambda word: (word, 1)) \
     .reduceByKey(lambda a, b: a+b)
counts.pprint()

stream.start()
stream.awaitTermination()

当我运行上述程序时，它会在终端上显示输出：

16/10/24 15:27:20错误TaskSetManager:阶段0.0中的任务0失败1次；

Answer 1

我可以确认修复是验证依赖项。

我的问题与SpringBoot/Java有关。对于Python项目，请查看使用pip install pipdeptree命令：https://www.geeksforgeeks.org/dependency-tree-of-a-python-module

对我来说，我升级了我的SpringBoot版本，它改变了我的SpringKafka测试intern包括kafka 2.13的版本。最终包括scala libs的x。

我在项目的构建文件中使用了mvn依赖项：Tree，并搜索了2.12，以找到旧的依赖项来自何处。示例依赖树（删除不相关的依赖）。注意org.apache.kafka：kafka_2.11： jar：0.10.0.0包含在my.company.riptide.api： Ness-logger： jar： 1.0.0但是org.springframework.kafka： sping-kafka-test： jar： 2.7.9包含了一个更新的版本

我的解决方案是将kafaka_2.11从我的ness记录器依赖中排除，如下所示：

<dependency>
  <groupId>my.company.riptide.api</groupId>
  <artifactId>ness-logger</artifactId>
  <version>1.0.0</version>
  <exclusions>
    <exclusion>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.11</artifactId>
    </exclusion>
  </exclusions>
</dependency>

mvn依赖项树的截断输出：

[INFO] +- my.company.riptide.api:ness-logger:jar:1.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-validation:jar:2.5.7:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:9.0.55:compile
[INFO] |  |  \- org.hibernate.validator:hibernate-validator:jar:6.2.0.Final:compile
[INFO] |  |     \- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] |  +- commons-io:commons-io:jar:2.7:compile
[INFO] |  +- my.company.eis:ness-logging-package:jar:4.0.1:compile
[INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  |  +- org.apache.avro:avro-compiler:jar:1.8.2:compile
[INFO] |  |  |  +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] |  |  |  \- joda-time:joda-time:jar:2.7:compile
[INFO] |  |  +- org.apache.kafka:kafka_2.11:jar:0.10.0.0:compile
[INFO] |  |  |  +- com.101tec:zkclient:jar:0.8:compile
[INFO] |  |  |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] |  |  \- com.netflix.hystrix:hystrix-core:jar:1.5.18:compile
[INFO] |  |     +- com.netflix.archaius:archaius-core:jar:0.4.1:compile
[INFO] |  |     \- io.reactivex:rxjava:jar:1.3.8:compile
[INFO] |  +- org.springframework.boot:spring-boot-loader-tools:jar:2.5.7:compile
[INFO] |  |  \- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] |  \- my.company.riptide.springboot:graceful-shutdown:jar:1.0.2:compile

[INFO] +- org.springframework.kafka:spring-kafka:jar:2.7.9:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.3.13:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.3.13:compile
[INFO] |  +- org.springframework.retry:spring-retry:jar:1.3.1:compile
[INFO] |  |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:2.7.2:compile
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.4.5-6:compile
[INFO] |  |  +- org.lz4:lz4-java:jar:1.7.1:compile
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.1.7.7:compile
[INFO] |  \- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:2.2:compile
[INFO] +- io.cucumber:cucumber-spring:jar:7.0.0:test
[INFO] |  \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] +- io.cucumber:cucumber-core:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-gherkin:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-gherkin-messages:jar:7.0.0:test
[INFO] |  +- io.cucumber:messages:jar:17.1.1:test
[INFO] |  +- io.cucumber:tag-expressions:jar:4.0.2:test
[INFO] |  +- io.cucumber:cucumber-expressions:jar:13.0.1:test
[INFO] |  +- io.cucumber:datatable:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-plugin:jar:7.0.0:test
[INFO] |  +- io.cucumber:docstring:jar:7.0.0:test
[INFO] |  +- io.cucumber:html-formatter:jar:17.0.0:test
[INFO] |  \- io.cucumber:create-meta:jar:6.0.1:test

[INFO] +- org.springframework.kafka:spring-kafka-test:jar:2.7.9:test
[INFO] |  +- org.apache.kafka:kafka-clients:jar:test:2.7.2:test
[INFO] |  +- org.apache.kafka:kafka-streams:jar:2.7.2:test
[INFO] |  |  +- org.apache.kafka:connect-json:jar:2.7.2:test
[INFO] |  |  |  \- org.apache.kafka:connect-api:jar:2.7.2:test
[INFO] |  |  \- org.rocksdb:rocksdbjni:jar:5.18.4:test
[INFO] |  +- org.apache.kafka:kafka-streams-test-utils:jar:2.7.2:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:2.7.2:test
[INFO] |  |  +- org.apache.kafka:kafka-raft:jar:2.7.2:test
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.13:jar:2.12.5:test
[INFO] |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.12.5:test
[INFO] |  |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:compile
[INFO] |  |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  |  +- org.scala-lang.modules:scala-collection-compat_2.13:jar:2.2.0:test
[INFO] |  |  +- org.scala-lang.modules:scala-java8-compat_2.13:jar:0.9.1:test
[INFO] |  |  +- org.scala-lang:scala-library:jar:2.13.3:compile
[INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.13.3:test
[INFO] |  |  +- com.typesafe.scala-logging:scala-logging_2.13:jar:3.9.2:test
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.5.9:compile
[INFO] |  |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.5.9:compile
[INFO] |  |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] |  |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.70.Final:compile
[INFO] |  |  \- commons-cli:commons-cli:jar:1.4:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:test:2.7.2:test
[INFO] |  \- org.junit.jupiter:junit-jupiter-api:jar:5.7.2:test
[INFO] |     +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |     \- org.junit.platform:junit-platform-commons:jar:1.7.2:test

Answer 2

Scala 2.10和2.11之间的集合API不同

<dependency>
                     <groupId>org.scala-lang</groupId>
                     <artifactId>scala-library</artifactId>
                     <version>2.10.6</version>
  </dependency>