转换列时Spark SQL-java.lang.unsupportedOperationException：empty.init

Question

当我试图对列执行强制转换时（从带有标题的逗号分隔的csv文件中读取），我得到了以下错误。

var df = spark.read.option("header","true").option("delimiter",",").csv("/user/sample/data")
df.withColumn("columnCast", expr("CAST(SaleAmount) AS LONG")).count

df.withColumn("columnCast", expr("CAST(NULL) AS LONG")).count

Answer 1

可以使用Column的cast方法进行强制转换：

import spark.implicits._

val df = spark.sparkContext.parallelize(1 to 10).toDF("col1")
val casted = df.withColumn("test", lit(null).cast("string"))
               .withColumn("testCast", $"test".cast("long"))
casted.show()
casted.printSchema()

结果：

+----+----+--------+
|col1|test|testCast|
+----+----+--------+
|   1|null|    null|
|   2|null|    null|
|   3|null|    null|
|   4|null|    null|
|   5|null|    null|
|   6|null|    null|
|   7|null|    null|
|   8|null|    null|
|   9|null|    null|
|  10|null|    null|
+----+----+--------+

root
 |-- col1: integer (nullable = false)
 |-- test: string (nullable = true)
 |-- testCast: long (nullable = true)