水槽在一行2048个字符后添加换行符

Question

我有一个在 Ubuntu 工作站上运行的 Flume 1.5 代理，它从各种设备收集日志并将日志重新格式化为具有很长行的逗号分隔文件。在收集和重新格式化日志后，它们被放入假脱机目录中，Flume 代理将日志文件发送到运行 Flume 代理的 Hadoop 服务器，以接受日志文件并将它们放在 HDFS 目录中。

除了当Flume将文件发送到HDFS目录时，每行每2048个字符后有换行符之外，一切都正常。

下面是我的水槽配置文件。是否有设置可以告诉水槽不要插入换行符？

#On Ubuntu Workstation
#list sources, sinks and channels in the agent
agent.sources = axon_source
agent.channels = memorychannel
agent.sinks = AvroOut

#define flow
agent.sources.axon_source.channels = memorychannel
agent.sinks.AvroOut.channel = memorychannel
agent.channels.memorychannel.type = memory
agent.channels.memorychannel.capacity = 100000

#source
agent.sources.axon_source.type = spooldir
agent.sources.axon_source.spoolDir = /home/ubuntu/workspace/logdump
agent.sources.axon_source.decodeErrorPolicy = ignore

#avro out
agent.sinks.AvroOut.type = avro
agent.sinks.AvroOut.hostname = 172.31.12.221
agent.sinks.AvroOut.port = 41415
agent.sinks.AvroOut.maxIoWorkers = 2


------------------------------------------------------------


#On Hadoop Server
agent.sources = AvroIn
agent.sources.AvroIn.type = avro
agent.sources.AvroIn.bind = 172.31.131.1
agent.sources.AvroIn.port = 41415
agent.sources.AvroIn.channels = MemChan1

agent.channels = MemChan1
agent.channels.MemChan1.type = memory
agent.channels.MemChan1.capacity = 100000

agent.sinks = HDFSSink
agent.sinks.HDFSSink.type = hdfs
agent.sinks.HDFSSink.channel = MemChan1
agent.sinks.HDFSSink.hdfs.path = /Logs/%Y%m/
agent.sinks.HDFSSink.hdfs.filePrefix = axoncapture
agent.sinks.HDFSSink.hdfs.fileSuffix = .log
agent.sinks.HDFSSink.hdfs.minBlockReplicas = 1
agent.sinks.HDFSSink.hdfs.rollCount = 0
agent.sinks.HDFSSink.hdfs.rollSize = 314572800
agent.sinks.HDFSSink.hdfs.writeFormat = Text
agent.sinks.HDFSSink.hdfs.fileType = DataStream
agent.sinks.HDFSSink.hdfs.useLocalTimeStamp = True

Answer 1

找到了我问题的答案:

LINE反序列化器的默认maxLineL的长度是2048：http://archive.cloudera.com/cdh5/cdh/5/flume-ng/FlumeUserGuide.html#line

我将该行添加到我的flume.conf文件中并修复了问题：agent.sources.axon_source.deserializer.maxLineL的长度=60000