Spark 1.4丢失的Kafka库

Question

我正在尝试运行一个在Spark1.3.1中完美工作的Python spark脚本。我已经下载了Spark1.4并尝试运行该脚本，但它一直在说

>

在spark-submit命令中将Kafka库及其依赖项包含为

$ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka:1.4.0 ...

从Maven Central http://search.Maven.org/下载工件的JAR，Group Id=org.apache.spark，artifact Id=spark-streaming-kafka-assembly，Version=1.4.0。然后，将jar包含在spark-submit命令中，作为

$ bin/spark-submit --jars <spark-streaming-kafka-assembly.jar> ...

我在submit命令中显式引用了JAR，并将JAR添加为

/opt/spark/spark-1.4.0-bin-hadoop2.6/bin/spark-submit --jars spark-streaming_2.10-1.4.0.jar,spark-core_2.10-1.4.0.jar,spark-streaming-kafka-assembly_2.10-1.4.0.jar,kafka_2.10-0.8.2.1.jar,kafka-clients-0.8.2.1.jar,spark-streaming-kafka-assembly_2.10-1.4.0.jar /root/SparkPySQLNew.py

它还说，当应用程序启动时，它已经添加了它们，为什么它没有找到它们呢？

15/07/08 05:44:37 INFO spark.SparkContext: Added JAR file:/root/spark-streaming_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-streaming_2.10-1.4.0.jar with timestamp 1436334277792
15/07/08 05:44:37 INFO spark.SparkContext: Added JAR file:/root/spark-core_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-core_2.10-1.4.0.jar with timestamp 1436334277919
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/spark-streaming-kafka-assembly_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-streaming-kafka-assembly_2.10-1.4.0.jar with timestamp 1436334278295
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/kafka_2.10-0.8.2.1.jar at http://192.168.134.138:49637/jars/kafka_2.10-0.8.2.1.jar with timestamp 1436334278353
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/kafka-clients-0.8.2.1.jar at http://192.168.134.138:49637/jars/kafka-clients-0.8.2.1.jar with timestamp 1436334278357
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/spark-streaming-kafka-assembly_2.10-1.4.0.jar at http://192.168.134.138:49637/jars/spark-streaming-kafka-assembly_2.10-1.4.0.jar with timestamp 1436334278665
15/07/08 05:44:38 INFO spark.SparkContext: Added JAR file:/root/spark-streaming-kafka-assembly_2.10-1.4.0-sources.jar at http://192.168.134.138:49637/jars/spark-streaming-kafka-assembly_2.10-1.4.0-sources.jar with timestamp 1436334278666               

我知道我已经加入了很多，我一开始就加入了一个，然后最后把它们都加进去了。

Answer 1

我怀疑每个版本的spark的确切答案都不一样，但是基于这个HCC线程，下面的内容似乎可以帮助其他人：

spark-submit --jars spark-assembly-1.5.2.2.3.4.7-4-hadoop2.7.1.2.3.4.7-4.jar,spark-streaming-kafka-assembly_2.10-1.6.1.jar 

乍一看，它有一个spark-streaming-kafka-assembly jar，而您提交的是两个。