Spark sql查询执行失败，org.apache.parquet.io.ParquetDecodingException

Question

我正在使用spark-submit（集群模式）在spark sql中执行一个简单的create table查询。正在接收org.apache.parquet.io.ParquetDecodingException。我可以从internet上获得关于这个问题的一些细节，其中一个建议是添加配置spark.sql.parquet.writelecacyformat=true。添加此设置后，该问题仍然存在。

spark.sql("""
CREATE TABLE TestTable
 STORED AS PARQUET 
    AS 
SELECT Col1, 
       Col2, 
       Col3 
FROM Stable""")

错误说明：

Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file maprfs:///path/disputer/1545555-r-00000.snappy.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:461)
at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:219)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:109)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:186)
... 13 more
Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableLong cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableInt

Spark配置文件：

spark.driver.memory=10G
spark.executor.memory=23G 
spark.executor.cores=3
spark.executor.instances=100  
spark.dynamicAllocation.enabled=false      
spark.yarn.preserve.staging.files=false  
spark.yarn.executor.extraJavaOptions=-XX:MaxDirectMemorySize=6144m    
spark.sql.shuffle.partitions=1000
spark.shuffle.service=true  
spark.yarn.maxAppAttempts=1  
spark.broadcastTimeout=36000  
spark.debug.maxToStringFields=100  
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2  
spark.network.timeout=600s  
spark.sql.parquet.enableVectorizedReader=false
spark.scheduler.listenerbus.eventqueue.capacity=200000  
spark.driver.memoryOverhead=1024  
spark.yarn.executor.memoryOverhead=5120  
spark.executor.extraJavaOptions=-XX:+UseG1GC  
spark.driver.extraJavaOptions=-XX:+UseG1GC

Answer 1

此问题是由于禁用spark.sql.parquet.enablevectorizedreader而发生的。enableVectorizedReader=true解决了这个问题。

有关详细信息，请访问https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-vectorized-parquet-reader.html