如何优雅地阻止水槽代理人

Question

许多网站建议在停止水槽代理时使用kill -9。

然而，当我用kill-9停止代理时，HDFS接收器文件将永远保持打开状态（如*.tmp）。

#Name the components on this agent
agent.sources = r1
agent.sinks = k1
agent.channels = c1

#Configure the Kafka Source
agent.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.r1.batchSize = 1000
agent.sources.r1.batchDurationMillis = 3000
agent.sources.r1.kafka.bootstrap.servers = <server1>:6667,<server2>:6667
agent.sources.r1.kafka.topics = 1-agent1-thread
agent.sources.r1.kafka.consumer.group.id = flume_agent_thread

#Describe the sink
agent.sinks.k1.type = hdfs
agent.sinks.k1.hdfs.path = /user/flume/kafka-data/1-agent1-thread/%y%m%d/%H
agent.sinks.k1.hdfs.filePrefix = 1-agent1-thread

#Describing sink with the problem of Encoding
agent.sinks.k1.hdfs.fileType = DataStream
agent.sinks.k1.hdfs.writeFormat = Text

#Describing sink with the problem of many hdfs files
### Roll a file after certain amount of events occurs  ###
agent.sinks.k1.hdfs.rollInterval = 0
agent.sinks.k1.hdfs.rollSize = 0
agent.sinks.k1.hdfs.rollCount = 10000
agent.sinks.k1.hdfs.batchSize = 100
agent.sinks.k1.hdfs.idleTimeout = 300
agent.sinks.k1.hdfs.closeTries = 0
agent.sinks.k1.hdfs.retryInterval = 200

#Use a channel which buffers events in memory
agent.channels.c1.type = memory
agent.channels.c1.capacity = 10000
agent.channels.c1.transactionCapacity = 1000

#Bind the source and sink to the channel
agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1

Answer 1

使用杀-TERM，这是杀死所有Hadoop相关服务的标准方法。

Flume将有一个SIGTERM处理程序（请参阅应用程序.java），它应该清理所有打开的文件。

ShutdownHook() -> stop() -> stopAllComponents()

杀-9只有在您尝试过杀-TERM并且水槽代理仍处于挂起状态时才应使用。