火花Kafka流媒体问题

Question

   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>

SparkConf sparkConf = new SparkConf().setAppName("KafkaSparkTest");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.addJar("/home/test/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.0.2/spark-streaming-kafka_2.10-1.0.2.jar");
JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(5000)); 

它没有任何错误，我得到以下错误时，我运行火花提交，任何帮助都非常感谢。谢谢你抽出时间。

bin/spark-submit --class "KafkaSparkStreaming" --master local[4] try/simple-project/target/simple-project-1.0.jar

线程“main”java.lang.noClassDeffounderror：org/apache/spark/streaming/kafka/kafkautils在kafkasparkstreaming.sparkstreamingtest(kafkasparkstreaming.java:40)在kafkasparkstreaming.main(kafkasparkstreaming.java:23)在sun.reflect.nativeMethodAccessorimpl.invoke0（原生方法）在sun.reflect.nativeMethodAccessorimpl.invoke（Method.java:606)在org.apache.spark.deploy.sparksubmit$.launch（sparksubmit.scala:303)在org.apache.spark.deploy.sparksubmit$.main（sparksubmit.scala:55)在org.apache.spark.deploy.sparksubmit.main（sparksubmit.scala）由:java.lang.classNotfoundexception:org.apache.spark.deploy.sparksubmit.scala（

Answer 1

我遇到了同样的问题，我通过构建具有依赖关系的jar来解决它。

>

删除代码中的“sc.addjar（）”。

将下面的代码添加到pom.xml中

<build>
    <sourceDirectory>src/main/java</sourceDirectory>
    <testSourceDirectory>src/test/java</testSourceDirectory>
    <plugins>
      <!--
                   Bind the maven-assembly-plugin to the package phase
        this will create a jar file without the storm dependencies
        suitable for deployment to a cluster.
       -->
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <archive>
            <manifest>
              <mainClass></mainClass>
            </manifest>
          </archive>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
</build>