在Scala中更改任何Spark sql StructType的所有元素的可空属性的常用方法

Question

是否有一种通用方法来更改任何指定StructType的所有元素的可空属性？它可能是嵌套的StructType。

我看到@eliasah将其标记为与Spark Dataframe列可空属性更改重复。但它们是不同的，因为它无法解决层次结构/嵌套结构类型，该答案仅适用于一个级别。

例如：

 root
 |-- user_id: string (nullable = false)
 |-- name: string (nullable = false)
 |-- system_process: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- timestamp: long (nullable = false)
 |    |    |-- process: string (nullable = false)
 |-- type: string (nullable = false)
 |-- user_process: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- timestamp: long (nullable = false)
 |    |    |-- process: string (nullable = false)

我想将nullalbe更改为true to all elements，结果应该是：

 root
 |-- user_id: string (nullable = true)
 |-- name: string (nullable = true)
 |-- system_process: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- timestamp: long (nullable = true)
 |    |    |-- process: string (nullable = true)
 |-- type: string (nullable = true)
 |-- user_process: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- timestamp: long (nullable = true)
 |    |    |-- process: string (nullable = true)

为了方便测试，随附了StructType的JSON模式示例：

val jsonSchema="""{"type":"struct","fields":[{"name":"user_id","type":"string","nullable":false,"metadata":{}},{"name":"name","type":"string","nullable":false,"metadata":{}},{"name":"system_process","type":{"type":"array","elementType":{"type":"struct","fields":[{"name":"timestamp","type":"long","nullable":false,"metadata":{}},{"name":"process_id","type":"string","nullable":false,"metadata":{}}]},"containsNull":false},"nullable":false,"metadata":{}},{"name":"type","type":"string","nullable":false,"metadata":{}},{"name":"user_process","type":{"type":"array","elementType":{"type":"struct","fields":[{"name":"timestamp","type":"long","nullable":false,"metadata":{}},{"name":"process_id","type":"string","nullable":false,"metadata":{}}]},"containsNull":false},"nullable":false,"metadata":{}}]}"""
DataType.fromJson(jsonSchema).asInstanceOf[StructType].printTreeString()

Answer 1

最后得出以下两种解决方案：

>

技巧一先替换字符串，然后从JSON字符串创建结构类型实例

DataType.fromJson(schema.json.replaceAll("\"nullable\":false", "\"nullable\":true")).asInstanceOf[StructType]

递归方法

  def updateFieldsToNullable(structType: StructType): StructType = {
    StructType(structType.map(f => f.dataType match {
      case d: ArrayType =>
        val element = d.elementType match {
          case s: StructType => updateFieldsToNullable(s)
          case _ => d.elementType
        }
        f.copy(nullable = true, dataType = ArrayType(element, d.containsNull))
      case s: StructType => f.copy(nullable = true, dataType = updateFieldsToNullable(s))
      case _ => f.copy(nullable = true)
    })
    )
  }