火花流集成水槽

Question

我遵循火花流水槽集成的指导。但我最终无法获得任何事件。（https://spark.apache.org/docs/latest/streaming-flume-integration.html）谁能帮我分析一下？在烟雾中，我创建了“avro_flume.conf”的文件，如下所示：

a1.sources = r1
a1.channels = c1
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 123.57.54.113
a1.sources.r1.port = 4141

a1.sinks = k1
a1.sinks.k1.type = avro

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = 123.57.54.113
a1.sinks.k1.port = 6666
a1.sources = r1
a1.sinks = spark
a1.channels = c1

在文件中，123.57.54.113是本地主机的ip。

flume-ng agent -c . -f conf/avro_spark.conf -n a1 Start Spark-streaming

bin/run-example org.apache.spark.examples.streaming.FlumeEventCount 123.57.54.113 6666

flume-ng avro-client -c . -H 123.57.54.113 -p 4141 -F test/log.01

最后，根本没有任何事件。

怎么了？谢谢！

Answer 1

我在标题“将源和汇绑定到通道”下看到“a1.sinks=spark”。但是名为“spark”的接收器没有在配置中的其他地方定义。你是在尝试“https://spark.apache.org/docs/latest/streaming-flume-integration.html"?

如果您正在尝试方法1，请尝试删除行“a1.sinks = spark”。

对于方法 2，请使用以下模板：

agent.sinks = spark
agent.sinks.spark.type = org.apache.spark.streaming.flume.sink.SparkSink
agent.sinks.spark.hostname = <hostname of the local machine>
agent.sinks.spark.port = <port to listen on for connection from Spark>
agent.sinks.spark.channel = memoryChannel