使用bucketBy的Spark架构与配置单元不兼容

Question

我正在使用Spark Version2.3使用Bucketby编写和保存数据。

表在配置单元中创建，但没有使用正确的模式。我无法从配置单元表中选择任何数据。

(DF.write
   .format('orc')
   .bucketBy(20, 'col1')
   .sortBy("col2")
   .mode("overwrite")
   .saveAsTable('EMP.bucketed_table1'))

hive> desc EMP.bucketed_table1;
OK
col                     array<string>           from deserializer

如何将数据记录保存并写入可稍后查看的配置单元表？

Answer 1

虽然Spark（至少在<=2.4版本中）不直接支持Hive的bucketing格式，但可以使用SparkSQL将数据加载到Hive中，从而使Spark输出Hive可读的bucketing数据；按照您的示例，它将如下所示：

//enable Hive support when creating/configuring the spark session
val spark = SparkSession.builder().enableHiveSupport().getOrCreate()

//register DF as view that can be used with SparkSQL
DF.createOrReplaceTempView("bucketed_df")

//create Hive table, can also be done manually on Hive
val createTableSQL = "CREATE TABLE bucketed_table1 (col1 int, col2 string) CLUSTERED BY col1 INTO 20 BUCKETS STORED AS PARQUET"
spark.sql(createTableSQL)

//load data from DF into Hive, output parquet files will be bucketed and readable by Hive
spark.sql("INSERT INTO bucketed_table1 SELECT * FROM bucketed_df")