数据流DoFn将不使用PipelineOptions序列化

Question

我试图将PipelineOptions接口传递给dataflow DoFn，以便DoFn可以配置一些它需要重新实例化的不可序列化的东西，但是当我告诉dataflow保存我的PipelineOptions子类的实例时，它似乎无法序列化DoFn。我需要对Options接口做什么才能使其正确序列化吗？

public interface Options 
extends BigtableOptions, BigtableScanOptions, OfflineModuleOptions, Serializable {...}

DoFn定义

public class RunEventGeneratorsDoFn extends DoFn<...,...> {
    private OfflinePipelineRunner.Options options;
....
}

选项未标记时的序列化异常瞬态

Exception in thread "main" java.lang.IllegalArgumentException: unable to serialize [my DoFn]
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:54)
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.clone(SerializableUtils.java:91)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$Bound.<init>(ParDo.java:720)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$Unbound.of(ParDo.java:678)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$Unbound.access$000(ParDo.java:596)
    at com.google.cloud.dataflow.sdk.transforms.ParDo.of(ParDo.java:563)
    at com.google.cloud.dataflow.sdk.transforms.ParDo.of(ParDo.java:558)
    at [dofn instantiation line]
Caused by: java.io.NotSerializableException: com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:50)
    ... 7 more

Answer 1

实际的管道选项对象不应作为字段包含在特定的dofn或ptransform中。相反，传入要访问的特定选项的值。

有关更多上下文“如何在Beam2.0中的复合PTransform中获取PipelineOptions？”，请参见此问题。