无法在java中读取SparkSQL中的文件

Question

请帮帮我，我是一个完全的傻瓜，当涉及到火花和Haddop一般。我的代码如下所示：

public static void main(String[] args) throws IOException {

    String[] jars = {"D:\\customJars\\sparky.jar","D:\\customJars\\guava-19.0.jar"};
      System.setProperty("hadoop.home.dir", "D:\\hadoop-common-2.2.0-bin-master");
        SparkConf sparkConf = new SparkConf().setAppName("com.nucleus.spark.MlibPOC")
                .setMaster("spark://10.1.50.165:7077")
                .setJars(jars);


    JavaSparkContext jsc = new JavaSparkContext(sparkConf);
    SQLContext sqlContext = new SQLContext(jsc);


   DataFrame df = sqlContext.read().json("src/com/nucleus/spark/generated.json");


}

我的Spark集群作为一个集群部署在10.1.50.165中，运行RHEL6，当我运行这段简单的代码时，在尝试读取json文件时，我遇到了如下异常：

Answer 1

这对我在Spark1.6上起作用：

    scala> val jtex = sqlContext.read.json("file:///opt/test.json")
    jtex: org.apache.spark.sql.DataFrame = [_corrupt_record: string, age: string, id: string, name: string]

    scala> val jtex = sqlContext.read.format("json").option("samplingRatio", "1.0").load("file:///opt/test.json")
    jtex: org.apache.spark.sql.DataFrame = [age: string, id: string, name: string]

    scala> jtex.show()
    +---+----+-------+
    |age|  id|   name|
    +---+----+-------+
    | 25|1201| satish|
    | 28|1202|krishna|
    | 39|1203|  amith|
    | 23|1204|  javed|
    | 23|1205| prudvi|
    +---+----+-------+