找不到Scala清单Spark流

Question

我对Spark Streaming是新手，从Spark Streaming我使用Kafkautils创建了一个直接到Kafka的流。如下所示

jssc = new JavaStreamingContext(sparkConf, Durations.seconds(KafkaConfig.getInstance().getBatchDuration()));

HashMap<String, String> kafkaParams = new HashMap<>();

kafkaParams.put(KafkaConstants.ZOOKEEPER_CONNECTION_STRING, zookeeperHost + ":" + zookeeperPort);
kafkaParams.put(KafkaConstants.METADATA_BROKER_LIST_STRING, bootstrapHost  + ":" + bootstrapPort);
kafkaParams.put(KafkaConstants.GROUP_ID_STRING, groupId);

HashSet<String> topicSet = new HashSet<String>();
topicSet.add(topic);

JavaPairInputDStream<String, String> topicStream = KafkaUtils.createDirectStream(jssc, String.class,
        String.class, StringDecoder.class, StringDecoder.class, kafkaParams, topicSet);

JavaDStream<String> topicMessages = topicStream.map(Tuple2::_2);
topicMessages.print()

当我试图运行该作业时，它正抛出以下错误

下面是我的pom.xml

<!-- Scala version -->
<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.10.3</version>
</dependency>

<!--Spark Core -->
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.1.1</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-8_2.11 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    <version>2.1.1</version>
</dependency>

<!--Spark Streaming -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>2.1.1</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.10 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-hive_2.11</artifactId>
    <version>2.1.1</version>
</dependency>

<!--Kafka -->
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.11</artifactId>
    <version>0.8.2.1</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>2.1.1</version>
</dependency>

请让我知道如何解决这个问题。

Answer 1

您已经定义了Scala version->2.10.3，并且编写了Artifactid和2.11。对于每个依赖项，您应该将artifactID更改为2.10，然后它就可以工作了。