Spark:数据帧Flatten中的嵌套数据结构

Question

我需要展平一个数据帧，以便将其与Spark（Scala）中的另一个数据帧连接起来。

基本上，我的2个数据帧有以下模式:

数据流1

root
|-- field1: string (nullable = true)
|-- field2: long (nullable = true)
|-- field3: long (nullable = true)
|-- field4: long (nullable = true)
|-- field5: integer (nullable = true)
|-- field6: timestamp (nullable = true)
|-- field7: long (nullable = true)
|-- field8: long (nullable = true)
|-- field9: long (nullable = true)
|-- field10: integer (nullable = true)

DF2

root
|-- field1: long (nullable = true)
|-- field2: long (nullable = true)
|-- field3: string (nullable = true)
|-- field4: integer (nullable = true)
|-- field5: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- field6: long (nullable = true)
|    |    |-- field7: integer (nullable = true)
|    |    |-- field8: array (nullable = true)
|    |    |    |-- element: struct (containsNull = true)
|    |    |    |    |-- field9: string (nullable = true)
|    |    |    |    |-- field10: integer (nullable = true)
|-- field11: timestamp (nullable = true)

老实说，我不知道如何使DF2变平。最后，我需要连接DF.field4 = DF2.field9上的2个数据帧

我用的是2.1.0

我的第一个想法是使用爆炸，但在Spark 2.1.0中已经被否决了，有人能给我一点提示吗？

Answer 1

我的错误分解功能在Spark 2.1.0中org.apache.spark.sql包中的functions.explode下仍然可用

谢谢

您可以在下面找到代码:

val DF2Exploded1 = DF2.select(DF2("*"), functions.explode(DF2("field5"))
                      .alias("field5_exploded"))

val DF2Exploded2 = DF2Exploded1.select(DF2Exploded1("*"), functions.explode(DF2Exploded1("field5_exploded.field8"))
                               .alias("field8_exploded"))