Spark Kafka流媒体问题

Question

我正在使用Maven

我添加了以下依赖项

   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>

我还在代码中添加了jar

SparkConf sparkConf = new SparkConf().setAppName("KafkaSparkTest");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.addJar("/home/test/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.0.2/spark-streaming-kafka_2.10-1.0.2.jar");
JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(5000));

它完全可以正常工作，没有任何错误，在通过spark-submit提交时出现以下错误，非常感谢您的帮助。谢谢你的时间。

bin/spark-submit --class "KafkaSparkStreaming" --master local[4] try/simple-project/target/simple-project-1.0.jar

线程“主要”
java.lang.NoClassDefFoundError中的异常：sun.reflect处的KafkaSparkStreaming.sparkStreamingTest（KafkaSparkStreaming.java:40）处的org
/ apache / spark / streaming / kafka /
KafkaUtils。位于sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）处的NativeMethodAccessorImpl.invoke0（本地方法），位于java.lang.reflect.Method.invoke（Method。
java：606）在org.apache.spark.deploy.SparkSubmit $
.launch（SparkSubmit.scala：303）在org.apache.spark.deploy.SparkSubmit $
.main（SparkSubmit.scala：55）在org.apache.spark
.deploy.SparkSubmit.main（SparkSubmit.scala）原因：java.lang.ClassNotFoundException：org.apache.spark.streaming.kafka。KafkaUtils
at java.net.URLClassLoader $ 1.run（URLClassLoader.java:366）

Answer 1

我遇到了同样的问题，我通过用依赖项构建jar解决了这个问题。

1. 在代码中删除“ sc.addJar（）”。

2. 将下面的代码添加到pom.xml

   <build>
   <sourceDirectory>src/main/java</sourceDirectory>
   <testSourceDirectory>src/test/java</testSourceDirectory>
   <plugins>
     <!--
                  Bind the maven-assembly-plugin to the package phase
       this will create a jar file without the storm dependencies
       suitable for deployment to a cluster.
      -->
     <plugin>
       <artifactId>maven-assembly-plugin</artifactId>
       <configuration>
         <descriptorRefs>
           <descriptorRef>jar-with-dependencies</descriptorRef>
         </descriptorRefs>
         <archive>
           <manifest>
             <mainClass></mainClass>
           </manifest>
         </archive>
       </configuration>
       <executions>
         <execution>
           <id>make-assembly</id>
           <phase>package</phase>
           <goals>
             <goal>single</goal>
           </goals>
         </execution>
       </executions>
     </plugin>
   </plugins>