Apache波束在数据流上的部署

Question

嗨，我已经创建了一个apache beam管道，测试了它，并在eclipse内部运行了它，包括本地和使用dataflow Runner。我可以在eclipse控制台中看到管道正在运行。e.控制台上的日志。

 SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in 
    [jar:file:/C:/Users/ThakurG/.m2/repository/org/slf4j/slf4j-
    jdk14/1.7.14/slf4j-jdk14-1.7.14.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/C:/Users/ThakurG/.m2/repository/org/slf4j/slf4j-nop/1.7.25/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]
    Jan 08, 2018 5:33:22 PM com.trial.apps.gcp.df.ReceiveAndPersistToBQ main
    INFO: starting the process...
    Jan 08, 2018 5:33:25 PM com.trial.apps.gcp.df.ReceiveAndPersistToBQ 
   createStream
    INFO: pipeline created::Pipeline#73387971
    Jan 08, 2018 5:33:27 PM com.trial.apps.gcp.df.ReceiveAndPersistToBQ main
    INFO: pie crated::Pipeline#73387971
    Jan 08, 2018 5:54:57 PM com.trial.apps.gcp.df.ReceiveAndPersistToBQ$1 apply
    INFO: Message received::1884408,16/09/2017,A,2007156,CLARK RUBBER FRANCHISING PTY LTD,A ,5075,6,Y,296,40467910,-34.868095,138.683535,66 SILKES RD,,,PARADISE,5075,0,7.4,5.6,18/09/2017 2:09,0.22
    Jan 08, 2018 5:54:57 PM com.trial.apps.gcp.df.ReceiveAndPersistToBQ$1 apply
    INFO: Payload from msg::1884408,16/09/2017,A,2007156,CLARK RUBBER FRANCHISING PTY LTD,A ,5075,6,Y,296,40467910,-34.868095,138.683535,66 SILKES RD,,,PARADISE,5075,0,7.4,5.6,18/09/2017 2:09,0.22
    Jan 08, 2018 5:54:57 PM com.trial.apps.gcp.df.ReceiveAndPersistToBQ$1 apply

这是我在cmd提示符中使用的maven命令，

`mvn compile exec:java -Dexec.mainClass=com.trial.apps.gcp.df.ReceiveAndPersistToBQ -Dexec.args="--project=analyticspoc-XXX --stagingLocation=gs://analytics_poc_staging --runner=DataflowRunner --streaming=true"`

这是我用来创建管道和设置选项的代码段。

java prettyprint-override">PipelineOptions options = PipelineOptionsFactory.create();

DataflowPipelineOptions dfOptions = options.as(DataflowPipelineOptions.class);
dfOptions.setRunner(DataflowRunner.class);
dfOptions.setJobName("gcpgteclipse");
dfOptions.setStreaming(true);

// Then create the pipeline.
Pipeline pipeL = Pipeline.create(dfOptions);

Answer 1

你能澄清一下“控制台显示它正在运行”和“无法使用Dataflow UI找到作业”到底是什么意思吗？

如果程序的输出打印消息：

To access the Dataflow monitoring console, please navigate to https://console.developers.google.com/project/.../dataflow/job/....

那么您的作业就在Dataflow服务上运行。一旦运行，关闭主程序并不会停止作业--主程序所做的只是周期性地轮询Dataflow服务以获取作业的状态和新的日志消息。按照打印的链接应该会把你带到Dataflow UI。

如果没有打印此消息，那么可能您的程序在实际启动数据流作业之前在某个地方卡住了。如果包含程序的输出，这将有助于调试。