使用yarn-cluster模式初始化SparkContext时出错

Question

我正在运行一个简单的代码来在hdfs上创建一个文件，并向其写入内容，然后关闭该文件。我可以在本地模式和纱线客户端模式下运行此代码。但是，当我用yarn-cluster模式运行相同的代码时，我在初始化sparkcontext时会遇到错误。

我使用的Scala代码如下所示：

object Main {
def main(args: Array[String]): Unit = {
val conf= new SparkConf()

var sparkContext =new SparkContext("yarn-cluster","testHdfsWrite",conf)
val uri = URI.create (hdfslocation); // hdfs location of my home directory
val confi = new Configuration();
val file = FileSystem.get(uri, confi);
val os = file.create(new Path(uri))
os.write("Hello World".getBytes("UTF-8"))
os.close()
 }
}

 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:411)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:147)
        at Main$.main(Main.scala:17)
        at Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
INFO SparkContext: Successfully stopped SparkContext




Answer 1

我也遇到过这个问题，并尝试使用以下方法来解决它：尝试一下，在程序的main（）方法中创建“spark context”（对于sql context相同），如下所示：

def main(args:array[String]):Unit={

  val conf = new SparkConf().setAppName("Simple Application")

  val sc = new SparkContext(conf)

  val sqlContext = new SQLContext(sc)

}并在下面导入：

spark-submit--master yarn-cluster--packages --class com.test.exampletest --jars