关于在集群(AWS)上运行spark作业的说明

Question

我有一个在AWS EC2机器上运行的HortonWorks集群，我想在上面运行一个使用spark streaming的spark工作，该工作将吞下tweet concernings《权力的游戏》。在尝试在集群上运行它之前，我确实在本地运行了它。代码正在工作，如下所示：

import org.apache.spark.streaming.{StreamingContext, Seconds}
import org.apache.spark.streaming.twitter._
import org.apache.spark.{SparkConf, SparkContext}

object Twitter_Stream extends App {

  val consumerKey = "hidden"
  val consumerSecret = "hidden"
  val accessToken = "hidden"
  val accessTokenSecret = "hidden"

  val sparkConf = new SparkConf().setAppName("GotTweets").setMaster("local[2]")

  val ssc = new StreamingContext(sparkConf, Seconds(1))

  val myStream = TwitterUtils.createStream(ssc, None, Array("#GoT","#WinterIsHere","#GameOfThrones"))

  val rddTweets = myStream.foreachRDD(rdd =>
  {
    rdd.take(10).foreach(println)
  })

  ssc.start()
  ssc.awaitTermination()
}

我的问题更确切地说是关于这段特定代码行：

val sparkConf = new SparkConf().setAppName("GotTweets").setMaster("local[2]")

17/07/24 11:53:42 INFO AppClient$ClientEndpoint:连接到主Spark://IP-Adress:7077...17/07/24 11:53:44警告AppClient$ClientEndpoint：未能连接到主IP-Adress：7077 java.io.ioException：未能连接到Spark：//IP-Adress：7077 at org.apache.spark.network.client.transportClientFactory.createClient（TransportClientFactory.java:216)at org.apache.spark.network.client.transportClientFactory.createClient（TransportClientFactory.java:167)at.netty.outbox$$anon$1.在org.apache.spark.rpc.netty.outbox$$anon$1调用(Outbox.scala:187)在java.util.concurrent.futuretask.run调用(Outbox.scala:183)在java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142)在java.util.concurrent.threadpoolexecutor.worker.run

Answer 1

要使用yarn运行spark应用程序，应该使用spark-submitusing--master yarn。不需要在scala源代码中使用setmaster。