PySpark 连接两个数据帧并更新嵌套结构

Question

我有两个火花数据帧。我想基于第二个更新第一个。

数据流1

[
  {"sku": "abc-abc-abc", 
   "prod_id": "ss-23235", 
   "salePrice": "2312"
  }, {
   "sku": "xyz-xyz-xyz", 
   "prod_id": "ss-13265", 
   "salePrice": "8312"
  }
]

df2

[{
  "sku": "abc-abc-abc", 
  "min_price": "678"
 },{
  "sku": "xyz-xyz-xyz", 
  "min_price": "7655"
 }
]

我想更新第一个数据帧（df1），如下所示

[
 {"sku": "abc-abc-abc", 
  "prod_id": "ss-23235", 
  "price": {
     "salePrice": "2312",
     "min_price": "678"
   },
 },
 {"sku": "xyz-xyz-xyz", 
  "prod_id": "ss-13265", 
  "price": {
     "salePrice": "8312",
      "min_price": "7655"
   }
 }
]

我不知道如何加入嵌套结构中的数据

Answer 1

import json
from pyspark.sql.types import ArrayType, IntegerType, StructType, StructField
from pyspark.sql import functions as F

Source1 = [
  {"sku": "abc-abc-abc", 
   "prod_id": "ss-23235", 
   "salePrice": "2312"
  }, {
   "sku": "xyz-xyz-xyz", 
   "prod_id": "ss-13265", 
   "salePrice": "8312"
  }
]
Source2 = [{
  "sku": "abc-abc-abc", 
  "min_price": "678"
 },{
  "sku": "xyz-xyz-xyz", 
  "min_price": "7655"
 }
]

df1 = sqlContext.read.json(sc.parallelize(Source1))
df1.show()
df2 = sqlContext.read.json(sc.parallelize(Source2))
df2.show()
df3 = df1.join(df2,'sku',how='inner').select(df1.sku,df1.prod_id,df1.salePrice,df2.min_price).withColumn('price',F.to_json(F.struct(df1.salePrice,df2.min_price))).drop('salePrice').drop('min_price')
df4 = df3.select('sku','prod_id','price').withColumn('Output',F.to_json(F.struct('sku','prod_id','price'))).drop('sku').drop('prod_id').drop('price').show(truncate=False)

---------------------------------------------------------------------------------------------------- |输出 | ---------------------------------------------------------------------------------------------------- |{"s ku"："abc-abc-abc"，"prod_id"："s-23235"，"价格"："{"销售价格"："2312"，"min_price"："678"}"}||{"sku"："xyz-xyz-xyz"，"prod_id"："ss-13265"，"价格"："{"salePrice"："8312"，"min_price"："7655"}"}| ----------------------------------------------------------------------------------------------------