Scala Spark-将数据从1个Dataframe复制到另一个具有嵌套模式的DF

Question

DF1-带数据的平面数据帧

+---------+--------+-------+                                                    
|FirstName|LastName| Device|
+---------+--------+-------+
|   Robert|Williams|android|
|    Maria|Sharpova| iphone|
+---------+--------+-------+

root
 |-- FirstName: string (nullable = true)
 |-- LastName: string (nullable = true)
 |-- Device: string (nullable = true)

DF2-具有相同列名的空数据帧

+------+----+
|header|body|
+------+----+
+------+----+

root
 |-- header: struct (nullable = true)
 |    |-- FirstName: string (nullable = true)
 |    |-- LastName: string (nullable = true)
 |-- body: struct (nullable = true)
 |    |-- Device: string (nullable = true)

DF2架构代码：

val schema = StructType(Array(
StructField("header", StructType(Array(
StructField("FirstName", StringType),
StructField("LastName", StringType)))), 
StructField("body", StructType(Array(
StructField("Device", StringType))))
))

带有来自DF1的数据的DF2将是最终输出。

需要为复杂模式的多列执行此操作并使其可配置。必须在不使用case类的情况下执行此操作。

方法#1-使用架构。字段。映射到映射DF1-

方法#2-创建一个新的DF并定义数据和模式？

方法#3-使用zip和map转换定义“select col as col”查询。。不知道这是否适用于嵌套（structtype）架构

我该如何继续这样做？

Answer 1

import spark.implicits._
import org.apache.spark.sql.functions._

val sourceDF = Seq(
  ("Robert", "Williams", "android"),
  ("Maria", "Sharpova", "iphone")
).toDF("FirstName", "LastName", "Device")

val resDF = sourceDF
  .withColumn("header", struct('FirstName, 'LastName))
  .withColumn("body", struct(col("Device")))
  .select('header, 'body)

resDF.printSchema
//  root
//  |-- header: struct (nullable = false)
//  |    |-- FirstName: string (nullable = true)
//  |    |-- LastName: string (nullable = true)
//  |-- body: struct (nullable = false)
//  |    |-- Device: string (nullable = true)

resDF.show(false)
//  +------------------+---------+
//  |header            |body     |
//  +------------------+---------+
//  |[Robert, Williams]|[android]|
//  |[Maria, Sharpova] |[iphone] |
//  +------------------+---------+