带有Flume的HDFS IO错误(hadoop 2.8)

Question

当我尝试通过Flume将流数据输入hadoop时，我收到以下错误。

我已经在flume/lib中创建了指向< code >的链接。hadoop/share/hadoop/中的jar文件

我仔细检查了URL，我认为它们都是正确的。想发帖子来获得更多的关注和反馈。

      2017-07-20 10:53:18,959 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN -org.apache.flume.sink.hdfs.HDFSEventSink.process HDFSEventSink.java:455)] HDFS IO error
      java.io.IOException: No FileSystem for scheme: hdfs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2798)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2809)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

这是水槽水槽配置

agent1.sinks.PurePathSink.type = hdfs
agent1.sinks.PurePathSink.hdfs.path = hdfs://127.0.0.1:9000/User/bts/pp 
agent1.sinks.PurePathSink.hdfs.fileType = DataStream
agent1.sinks.PurePathSink.hdfs.filePrefix = export
agent1.sinks.PurePathSink.hdfs.fileSuffix = .txt
agent1.sinks.PurePathSink.hdfs.rollInterval = 120
agent1.sinks.PurePathSink.hdfs.rollSize = 131072

核心网站. xml - Hadoop 2.8

<configuration>

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home1/tmp</value>
        <description>A base for other temporary directories</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://127.0.0.1:9000</value>
    </property>

    <property>
        <name>fs.file.impl</name>
        <value>org.apache.hadoop.fs.LocalFileSystem</value>
        <description>The FileSystem for file: uris.</description>
    </property>

    <property>
        <name>fs.hdfs.impl</name>
        <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
        <description>The FileSystem for hdfs: uris.</description>
    </property>

Answer 1

在我的例子中，将hdfs-jar文件从hadoop/hdfs复制到flume/lib解决了这个问题。

$ cp my_hadoop_path/share/hadoop/hdfs/*.jar my_flume_path/lib/

Answer 2

在我的例子中，我发现显式声明路径解决了这个问题。这与它捡到的是哪个罐子有关。

感谢@V.Bravo的回复。我没有使用发行版，而是建立了自己的集群

• 摩西

Answer 3

查看您的Flume Sink，似乎您不是在集群上运行这个，而是在本地主机上运行。

检查HDFS路径是否可访问:

agent1.sinks.PurePathSink.hdfs.path = hdfs://127.0.0.1:9000/User/bts/pp

端口号通常是8020(如果您使用的是Cloudera发行版)

请查看以下链接以了解错误复制和解决方案：[Cloudera Solved:FLUME IO error Problem]

https://community.cloudera.com/t5/Storage-Random-Access-HDFS/Flume-HDFS-IO-error-ConnectException/td-p/28157