使用Spark Scala以filename作为参数读取Excel文件时出错

Question

有人能帮我用Spark Scala读API读excel文件吗？我尝试安装com.crealytics:spark-excel2.11:0.13.1（从Maven）与Databricks Runtime 6.5和6.6（Apache Spark 2.4.5,Scala 2.11）集群，但只有在硬编码文件路径的情况下才能工作。

val df = spark.read
    .format("com.crealytics.spark.excel")
    .option("sheetName", "Listing_Attributed")
    .option("header", "true")
    .option("inferSchema", "false")
    .option("addColorColumns", "true") // Optional, default: false
    .option("badRecordsPath", Vars.rootSourcePath + "BadRecords/" + DataCategory)
    .option("dateFormat", "dd-MON-yy")
    .option("timestampFormat", "MM/dd/yyyy hh:mm:ss")
    .option("ignoreLeadingWhiteSpace",true)
    .option("ignoreTrailingWhiteSpace",true)
    .option("escape"," ")
    .load("/ABC/Test_Filename_6.12.20.xlsx")  // hard-coded path works...
//  .load(filepath)    //Filepath is a parameter and throws error, "java.io.IOException: GC overhead limit exceeded" (edited)

Answer 1

使用.option（“location”，inputPath)，如下所示

 val df = spark.read
        .format("com.crealytics.spark.excel")
        .option("sheetName", "Listing_Attributed")
        .option("header", "true")
        .option("location", inputPath)
        .load()