从apache beam pcollection返回什么内容以写入bigquery

Question

我正在阅读beam文档和一些stackoverflow问题/答案，以便了解如何向BigQuery编写pubsub消息。到目前为止，我已经有了获取protobuf消息并能够解码消息的工作示例。代码如下所示

(p
 | 'ReadData' >> apache_beam.io.ReadFromPubSub(topic=known_args.input_topic, with_attributes=True)
 | 'ParsePubsubMessage' >> apache_beam.Map(parse_pubsubmessage)
 )

class DecodedPubsubMessage:
    def __init__(self, attr, event):
        self.attribute_one = attr['attribute_one']
        self.attribute_two = attr['attribute_two']

        self.order_id = event.order.order_id
        self.sku = event.item.item_id
        self.triggered_at = event.timestamp
        self.status = event.order.status

Answer 1

您的类必须从DoFn继承并重载“process”方法，而不是在init上执行转换

在转换之后，您可以使用“return[obj]”或“yield obj”返回所需的输出PCollection