无法在HDP 2.0上运行Spark 1.0 SparkPi

Question

 ./bin/spark-submit --class org.apache.spark.examples.SparkPi     --master yarn-cluster --num-executors 3 --driver-memory 2g --executor-memory 2g --executor-cores 1 ./lib/spark-examples-1.0.0-hadoop2.2.0.jar 2

我有错误：

应用程序application_1404470405736_0044失败3次，原因是appattempt_1404470405736_0044_000003的容器退出，退出代码：1原因是：容器异常-启动：org.apache.hadoop.util.shell$exitcodeexception：在org.apache.hadoop.util.shell.runcommand(shell.java:464)在org.apache.hadoop.util.shell.run(shell.java:379)在org.apache.hadoop.util.shell$shellcommandexecutor.execute(shell.java:589)在）在java.lang.Thread.Run（Thread.java:744)处。如果尝试失败。应用程序失败。

未知/不支持的参数列表（--executor-memory,2048,--executor-cores,1,--num-executors,3）用法：org.apache.spark.deploy.yarn.applicationmaster[options]选项:
--JAR JAR_PATH应用程序JAR文件的路径（必需）--class CLASS_NAME应用程序主类的名称（必需）...bla-bla-bla

有什么想法吗？我怎么才能让它起作用呢？

Answer 1

我也有同样的问题。原因是hdfs中的spark-assembly.jar版本与当前的spark版本不同。

例如，org.apache.spark.deploy.yarn.client在hdfs版本中的params列表：

  $ hadoop jar ./spark-assembly.jar  org.apache.spark.deploy.yarn.Client --help
Usage: org.apache.spark.deploy.yarn.Client [options] 
Options:
  --jar JAR_PATH             Path to your application's JAR file (required in yarn-cluster mode)
  --class CLASS_NAME         Name of your application's main class (required)
  --args ARGS                Arguments to be passed to your application's main class.
                             Mutliple invocations are possible, each will be passed in order.
  --num-workers NUM          Number of workers to start (Default: 2)
  --worker-cores NUM         Number of cores for the workers (Default: 1). This is unsused right now.
  --master-class CLASS_NAME  Class Name for Master (Default: spark.deploy.yarn.ApplicationMaster)
  --master-memory MEM        Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb)
  --worker-memory MEM        Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
  --name NAME                The name of your application (Default: Spark)
  --queue QUEUE              The hadoop queue to use for allocation requests (Default: 'default')
  --addJars jars             Comma separated list of local jars that want SparkContext.addJar to work with.
  --files files              Comma separated list of files to be distributed with the job.
  --archives archives        Comma separated list of archives to be distributed with the job.

和最新安装的spark-assembly jar文件的相同帮助：

$ hadoop jar ./spark-assembly-1.0.0-cdh5.1.0-hadoop2.3.0-cdh5.1.0.jar org.apache.spark.deploy.yarn.Client
Usage: org.apache.spark.deploy.yarn.Client [options] 
Options:
  --jar JAR_PATH             Path to your application's JAR file (required in yarn-cluster mode)
  --class CLASS_NAME         Name of your application's main class (required)
  --arg ARGS                 Argument to be passed to your application's main class.
                             Multiple invocations are possible, each will be passed in order.
  --num-executors NUM        Number of executors to start (Default: 2)
  --executor-cores NUM       Number of cores for the executors (Default: 1).
  --driver-memory MEM        Memory for driver (e.g. 1000M, 2G) (Default: 512 Mb)
  --executor-memory MEM      Memory per executor (e.g. 1000M, 2G) (Default: 1G)
  --name NAME                The name of your application (Default: Spark)
  --queue QUEUE              The hadoop queue to use for allocation requests (Default: 'default')
  --addJars jars             Comma separated list of local jars that want SparkContext.addJar to work with.
  --files files              Comma separated list of files to be distributed with the job.
  --archives archives        Comma separated list of archives to be distributed with the job.

因此，我将Spark-Assembly.jar更新为hdfs，spark开始很好地工作