Spark Dataframe转换到的rdd,取出Array类型字段的数据的时候,采用
spark.sql(servicesql).rdd.map(r => {
val property = r.getAs[List[String]]("property")
})
会报错。
Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to scala.collection.immutable.List
at Spark$$anonfun$1.apply(testSpark.scala:100)
at Spark$$anonfun$1.apply(testSpark.scala:97)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
at scala.collection.AbstractIterator.to(Iterator.scala:1334)
应该默认是seq,而seq无法直接强制转换为List,需要函数。或者直接采用Seq,这样更加直接。
解决,设置为seq
val property = r.getAs[Seq[String]]("property")