数据流中带有时间戳的进程字段

Question

{u'date': u'2019-03-26T09:57:52Z', 'field1': value1, u'field2': u'value2', u'field3': u'value3', u'field4': u'value4',...}

| 'Window' >> beam.WindowInto(window.FixedWindows(1 * 10))

我需要一个自定义的WindowFn，或者应该怎么做？

Answer 1

您需要像这样指定自定义时间戳：

def custom_timestamp(message):
    # assuming that message is already parsed JSON (dict)
    import datetime as dt
    import apache_beam as beam
    ts = dt.datetime.strptime(message["date"], "%Y-%m-%dT%H:%M:%SZ")
    return beam.window.TimestampedValue(message, ts.timestamp())

后来呢：

| 'CustomTimestamp' >> beam.Map(custom_timestamp)
| 'Window' >> beam.WindowInto(window.FixedWindows(1 * 10))

您可以在下面找到完整的详细信息:https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements