Apache Flume .tmp强制终止启动后未清理的文件

Question

我看到了几个与此相关的问题，但不太明白我在寻找什么。我使用的是Flume 1.8.0，在Flume代理被强制终止后，我看到了.tmp文件。这些不会在水槽药剂重新启动时清理干净。有没有任何方法可以配置Flume代理来执行清理，或者这是我需要自己处理的事情（在Flume之外）？

这是我测试的水槽 conf 文件：

agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = hdfs1

# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = seq

# The channel can be defined as follows.
agent.sources.seqGenSrc.channels = memoryChannel

# Each sink's type must be defined
#agent.sinks.loggerSink.type = logger

#Specify the channel the sink should use
#agent.sinks.loggerSink.channel = memoryChannel
agent.sinks.hdfs1.type = hdfs
agent.sinks.hdfs1.hdfs.path = hdfs://localhost:9000/flume/
agent.sinks.hdfs1.hdfs.fileType = CompressedStream
agent.sinks.hdfs1.hdfs.codeC = bzip2
agent.sinks.hdfs1.hdfs.file.Prefix = myfiles-
agent.sinks.hdfs1.channel = memoryChannel

agent.sinks.hdfs1.hdfs.batchSize = 15
agent.sinks.hdfs1.hdfs.rollSize = 1048576
agent.sinks.hdfs1.hdfs.rollInterval = 20
agent.sinks.hdfs1.hdfs.rollCount = 0
agent.sinks.hdfs1.hdfs.writeFormat = Writable
agent.sinks.hdfs1.hdfs.useLocalTimeStamp = true
agent.sinks.hdfs1.hdfs.timeZone = UTC
agent.sinks.hdfs1.hdfs.callTimeout = 60000

agent.sinks.hdfs1.hdfs.idleTimeout = 0
agent.sinks.hdfs1.hdfs.maxOpenFiles = 1
agent.sinks.hdfs1.hdfs.retryInterval = 1

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100

使用上面的conf文件启动Flume代理。在它向HDFS写入了几个文件后，使用杀-9杀死它。检查hdfs目录。将有一个. tmp文件。如果我们重新启动Flume代理，它将恢复向HDFS写入数据，但它不会关闭/清理. tmp文件。

Answer 1

Flume HDFS接收器创建tmp文件，一旦达到大小、时间或事件数量的限制，它就会滚动文件，意味着关闭。tmp文件，并通过移除。tmp扩展。您需要使用< code>kill -15而不是< code>kill -9，以便flume代理可以正常退出，并在停止代理之前执行这些任务。