spark与kafka的集成，spark中的例外-提交一个jar

Question

我试着检查在Ubuntu下安装spark是否需要set HADOOP_HOME；但是，没有设置HADOOP_HOME，请仔细检查JAR的参数。

./bin/spark-submit --class "org.apache.spark.examples.streaming.JavaKafkaWordCount" --packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0 --master local[*] --jars ~/software/JavaKafkaWordCount.jar localhost:2181 test-consumer-group streams-plaintext-input 1

线程“main”java.lang.NullPointerException（位于org.apache.hadoop.fs.path.getName（Path.java:337)（位于org.apache.spark.deploy.deploy.deployls$.downloadfile（deploydencyutils.scala:136)（位于org.apache.spark.deploy.sparksubmit$$anonfun$preparesubmitEnvironment$7)apply(sparksubmit.scala:367)（位于

Answer 1

您的路径uri jar不可理解参见此dependencyutils.scala#L136

 /**
   * Download a file from the remote to a local temporary directory. If the input path points to
   * a local path, returns it with no operation.
   *
   * @param path A file path from where the files will be downloaded.
   * @param targetDir A temporary directory for which downloaded files.
   * @param sparkConf Spark configuration.
   * @param hadoopConf Hadoop configuration.
   * @param secMgr Spark security manager.
   * @return Path to the local file.
   */
  def downloadFile(
      path: String,
      targetDir: File,
      sparkConf: SparkConf,
      hadoopConf: Configuration,
      secMgr: SecurityManager): String = {
    require(path != null, "path cannot be null.")
    val uri = Utils.resolveURI(path)

    uri.getScheme match {
      case "file" | "local" => path
      case "http" | "https" | "ftp" if Utils.isTesting =>
        // This is only used for SparkSubmitSuite unit test. Instead of downloading file remotely,
        // return a dummy local path instead.
        val file = new File(uri.getPath)
        new File(targetDir, file.getName).toURI.toString
      case _ =>
        val fname = new Path(uri).getName()
        val localFile = Utils.doFetchFile(uri.toString(), targetDir, fname, sparkConf, secMgr,
          hadoopConf)
        localFile.toURI().toString()
    }
  }

在spark-submit中，如下所示更改参数

--jars/fullpath/javakafkaWordcount.jar而不是--jars~/software/javakafkaWordcount.jar