配置单元将数据配置为结构数组

Question

我试图找出一种在Hive中从平面源中选择数据并将其输出到一个名为struct的数组中的方法。这是我正在寻找的示例…

样本数据：

house_id,first_name,last_name
1,bob,jones
1,jenny,jones
2,sally,johnson
3,john,smith
3,barb,smith

所需的输出：

1   [{"first_name":"bob","last_name":"jones"},{"first_name":"jenny","last_name":"jones"}]
2   [{"first_name":"sally","last_name":"johnson"}]
3   [{"first_name":"john","last_name":"smith"},{"first_name":"barb","last_name":"smith"}]

我尝试了collect_list和collect_set，但是它们仅允许原始数据类型。关于如何在Hive中进行此操作有任何想法吗？

Answer 1

我会使用这个jar，它是的更好的实现collect（并需要复杂的数据类型）。

查询 ：

add jar /path/to/jar/brickhouse-0.7.1.jar;
create temporary function collect as 'brickhouse.udf.collect.CollectUDAF';

select house_id
  , collect(named_struct("first_name", first_name, "last_name", last_name))
from db.table
group by house_id

输出 ：

1   [{"first_name":"bob","last_name":"jones"}, {"first_name":"jenny","last_name":"jones"}]
2   [{"first_name":"sally","last_name":"johnson"}]
3   [{"first_name":"john","last_name":"smith"},{"first_name":"barb","last_name":"smith"}]