在Windows 7中运行Spark SQL连接Yarn-Client

Question

错误：

15/09/04 14:06:14 INFO HttpServer: Starting HTTP Server
15/09/04 14:06:14 INFO Utils: Successfully started service 'HTTP file server' on port 56407.
15/09/04 14:06:14 INFO SparkEnv: Registering OutputCommitCoordinator
15/09/04 14:06:15 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/09/04 14:06:15 INFO SparkUI: Started SparkUI at http://169.254.196.102:4040
15/09/04 14:06:18 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/09/04 14:06:20 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:22 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:24 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:26 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/09/04 14:06:28 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Scala代码：

import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkConf, SparkContext}


/**
 * Created by gfp2ram on 9/3/2015.
 */

object HiveConnect {
  def main(args: Array[String]) {
    System.setProperty("hadoop.home.dir", "C:\\hadoop\\hadoop-common-2.2.0-bin-master\\")
//    val conf = new SparkConf().setAppName("HiveConnect").setMaster("yarn-client").set("spark.yarn.access.namenodes","hdfs://127.0.0.1:8020")
    val conf = new SparkConf().setAppName("HiveConnect").setMaster("yarn-client")
    val sc = new SparkContext(conf)
    val sqlContext = new HiveContext(sc)
    val df = sqlContext.sql("create table yahoo_orc_table (date STRING, open_price FLOAT, high_price FLOAT, low_price FLOAT, close_price FLOAT, volume INT, adj_price FLOAT) stored as orc")
    val yahoo_stocks = sc.textFile("hdfs://127.0.0.1:8020/tmp/yahoo_stocks.csv")
    val header = yahoo_stocks.first
    val data = yahoo_stocks.filter(_(0) != header(0))
    case class YahooStockPrice(date: String, open: Float, high: Float, low: Float, close: Float, volume: Integer, adjClose: Float)
    val stockprice = data.map(_.split(",")).map(row => YahooStockPrice(row(0), row(1).trim.toFloat, row(2).trim.toFloat, row(3).trim.toFloat, row(4).trim.toFloat, row(5).trim.toInt, row(6).trim.toFloat))
    println(stockprice)
    stockprice.registerTempTable("yahoo_stocks_temp")
    val results = sqlContext.sql("SELECT * FROM yahoo_stocks_temp")
    results.map(t => "Stock Entry: " + t.toString).collect().foreach(println)
    results.saveAsOrcFile("yahoo_stocks_orc")
    val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
    val yahoo_stocks_orc = sqlContext.orcFile("yahoo_stocks_orc")
    yahoo_stocks_orc.registerTempTable("orcTest")
    sqlContext.sql("SELECT * from orcTest").collect.foreach(println)
  }
}

Answer 1

我不能添加一个评论，由于缺乏声誉点。然而，从日志来看，spark似乎无法连接到资源管理器。

15/09/04 14:06:15信息SparkUI：在http://169.254.196.102:4040 15/09/04 14:06:18信息rmproxy：在/0.0.0.0:8032连接到ResourceManager

您能否ping并检查以确保resourcemanager可以从windows Mahine访问。