Flume不适用于HDFS

Question

我用两个节点设置水槽。我想将数据从slave01加载到hdfs。slave01:示例配置属性

 agent.sources = baksrc
agent.channels = memoryChannel
agent.sinks =avro-forward-sink 
agent.sources.baksrc.type = exec
agent.sources.baksrc.command = tail -F /root/hadoo/test/data.txt
agent.sources.baksrc.checkperiodic = 1000
agent.sources.baksrc.channels =memoryChannel 
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 10000
agent.sinks.avro-forward-sink.type = avro
agent.sinks.avro-forward-sink.hostname = master
agent.sinks.avro-forward-sink.port = 23004
agent.sinks.avro-forward-sink.channel = memoryChannel

master:示例配置属性

agent.sources = avrosrc
agent.sinks =hdfs-write 
agent.channels = memoryChannel
agent.sources.avrosrc.type =avro
agent.sources.avrosrc.bind =master
agent.sources.avrosrc.port =23004
agent.sources.avrosrc.channels=memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity =10000
agent.sinks.hdfs-write.type = hdfs
agent.sinks.hdfs-write.hdfs.path =hdfs://172.16.86.38:9000/flume/webdata
agent.sinks.hdfs-write.hdfs.rollInterval = 0   
agent.sinks.hdfs-write.hdfs.rollSize = 4000000  
agent.sinks.hdfs-write.hdfs.rollCount = 0   
agent.sinks.hdfs-write.hdfs.writeFormat = Text  
agent.sinks.hdfs-write.hdfs.fileType = DataStream  
agent.sinks.hdfs-write.hdfs.batchSize = 10  
agent.sinks.hdfs-write.channel=memoryChannel  

然后我运行一个shell脚本：如下所示：

#!/bin/sh
for i in {1..1000000}; do
    echo "test flume to Hbase $i" >>/root/hadoop/test/data.txt; 
    sleep 0.1; 
done

启动 flume： flume-ng agent --conf conf conf --conf-file example-conf.properties --name agent -dflume.root.logger=DEBUG，console 我在控制台上没有错误。

14/05/06 16:38:44 INFO source.AvroSource: Avro source avrosrc stopping: Avro source avrosrc: { bindAddress: master, port: 23004 }
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] DISCONNECTED
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] UNBOUND
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] CLOSED

但是我看不到hdfs中的文件，我的配置有问题吗？？我已经在master上测试过了，效果很好。

Answer 1

您使用哪个版本的水槽？
你设置HADOOP_HOME了吗？
Flume是否从HADOOP_HOME输出带有Hadoop jar的类路径？
如果您使用的是Apache Flume，那么一步一步： 1. 设置HADOOP_HOME
2. 编辑 Hadoop核心站点.xml，确保 namenode IP 正确。
3. 使用 HDFS 路径：agent.sinks.hdfs-write.hdfs.path =/flume/webdata
4. 启动水槽