火花拼花器读数误差

Question

我在一个Spark项目上工作，这里我有一个文件是在parquet格式，当我试图用java加载这个文件时，它给了我下面的错误。但是，当我用相同的路径在hive中加载相同的文件并编写查询select*from table_name时，它工作得很好，数据也很正常。关于这个问题，请帮助我。

java.io.ioException：无法读取页脚:java.lang.runtimeException：损坏的文件：页脚索引不在org.apache.parquet.hadoop.parquetFileReader.readAllFootersinParallel(ParquetFileReader.java:247)在org.apache.spark.sql.execution.datasources.parquet.parquetRelation$$AnonFun$28处的文件中。在org.apache.spark.execution.datasources.parquet.parquetRelations he.spark.rdd.rdd$$anonfun$mappartitions$1$$anonfun$apply$17.applic(rdd.scala:710)在org.apache.spark.rdd.mappartitionsrdd.compute(mappartitionsrdd.scala:38)在org.apache.spark.rdd.rdd.rdd.rdd.computeorreadcheckpoint(rdd.scala:300)在.spark.scheduler.resulttask.runtask(resulttask.scala:66)在org.apache.spark.scheduler.task.run(task.scala:88)在org.apache.spark.executor.executor$t在java.util.concurrent.ThreadPoolExecutor.runworker（ThreadPoolExecutor.java:1145)在java.util.concurrent.ThreadPoolExecutor$worker.run（ThreadPoolExecutor.java:615)在java.lang.thread.run（ThreadPoolExecutor.java.java.java.java.java:745)由：java.lang.thread.run（ThreadPoolExecutor.java:745)引起：eader$2.在org.apache.parquet.hadoop.parquetFileReader.java:237)调用(parquetFileReader.java:233)在java.util.concurrent.futureTask.run(FutureTask.java:262)调用(parquetFileReader.java:232)

Answer 1

You can try below options

1)  sqlContext.read.parquet("path") 

2) sqlContext.read.format(fileFormat)
.option("header", header) // Use first line of all files as header
.option("inferSchema", inferSchema) // Automatically infer data types
.load(source)

If your issue didn't resolved, please post the sample of code.