spark-salesforce连接器中的Null类型架构

Question

我有一个数据集 ，其中包含从Salesforce导入的48列：

Dataset<Row> df = spark.read()
  .format("com.springml.spark.salesforce")
  .option("username", prop.getProperty("salesforce_user"))
  .option("password", prop.getProperty("salesforce_auth"))
  .option("login", prop.getProperty("salesforce_login_url"))
  .option("soql", "SELECT "+srcCols+" from "+tableNm)
  .option("version", prop.getProperty("salesforce_version"))
  .load()

finalDS.coalesce(1).write().option("delimiter", "^").toString().text(hdfsExportLoaction);

Exception in thread "main" org.apache.spark.sql.AnalysisException: Try to map struct<Columns....>to Tuple1, but failed as the number of fields does not line up.;
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveDeserializer$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveDeserializer$$fail(Analyzer.scala:2320)

finalDS.map(row -> row.mkString(), Encoders.STRING()).write().option("delimiter", "^").text(hdfsExportLoaction);

finalDS.coalesce(1).write().mode(SaveMode.Overwrite).option("header", "true").option("delimiter", "^").option("nullValue", "").csv(hdfsExportLoaction+"/"+tableNm);

finalDS.na().fill("").coalesce(1).write().option("delimiter", "^").mode(SaveMode.Overwrite).csv(hdfsExportLoaction);

Exception in thread "main" java.lang.UnsupportedOperationException: CSV data source does not support null data type.

当试图以文本文件的形式写入时，可能会删除分隔符，或者只能将单个列写入文本文件的错误，
当试图以CSV的形式写入时，则不支持Null数据类型异常。

Answer 1

我认为您的数据集或数据表本身有问题。对我来说

df.coalesce(1).write.option("delimiter", "^").mode(SaveMode.Overwrite).csv("<path>")

这和预期的一样。它正确地用“^”分隔。我建议检查您的dataframe或DataSet的数据以及您正在对其进行的操作。在写入数据之前，使用一次df.count并查看其失败与否