通过Apache Flume将日志文件从本地文件系统移动到HDFS时出错

Question

我的本地文件系统中有日志文件，需要通过Apache Flume传输到HDFS。我在主目录中将以下配置文件另存为net.conf

NetcatAgent.sources = Netcat
NetcatAgent.channels = MemChannel
NetcatAgent.sinks= LoggerSink

# configuring source
NetcatAgent.sources.Netcat.type = netcat
    #type of conection is netcat
NetcatAgent.sources.Netcat.bind = localhost
    # bind to localhost
NetcatAgent.sources.Netcat.port=9999
    # localhost port number


# configuring sink
NetcatAgent.sinks.LoggerSink.type = logger
    #logger sends output to console

# Configuring Channel
NetcatAgent.channels.MemChannel.type = memory   
    #defines type of memory it is storing
NetcatAgent.channels.MemChannel.capacity = 10000   
    #how many events can be present
NetcatAgent.channels.MemChannel.transactionCapacity = 1000  
    #how many events it can handle at a time

# bind source and sink to channel
NetcatAgent.sources.Netcat.channels = MemChannel
NetcatAgent.sinks.LoggerSink.channel = MemChannel



#to run the file on console 
#flume-ng agent -n NetcatAgent -f net.conf

#on other terminal establish connection using
#telnet localhost 9999

在主目录中运行命令后Flume-ng agent-n NetcatAgent-fnet.conf

我得到了以下输出：

Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/home/samar/hadoop-3.3.1/bin/hadoop) for HDFS access
Info: Including Hive libraries found via () for Hive access
+ exec /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xmx20m -cp '/home/samar/flume/lib/*:/home/samar/hadoop-3.3.1/etc/hadoop:/home/samar/hadoop-3.3.1/share/hadoop/common/lib/*:/home/samar/hadoop-3.3.1/share/hadoop/common/*:/home/samar/hadoop-3.3.1/share/hadoop/hdfs:/home/samar/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/home/samar/hadoop-3.3.1/share/hadoop/hdfs/*:/home/samar/hadoop-3.3.1/share/hadoop/mapreduce/*:/home/samar/hadoop-3.3.1/share/hadoop/yarn:/home/samar/hadoop-3.3.1/share/hadoop/yarn/lib/*:/home/samar/hadoop-3.3.1/share/hadoop/yarn/*:/lib/*' -Djava.library.path=:/home/samar/hadoop-3.3.1/lib/native org.apache.flume.node.Application -n NetcatAgent -f net.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/samar/flume/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/samar/hadoop-3.3.1/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
    at java.base/jdk.internal.loader.Resource.getBytes(Resource.java:117)
    at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:797)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at com.google.common.collect.Sets.newHashSetWithExpectedSize(Sets.java:194)
    at com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:114)
    at com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:49)
    at com.google.common.collect.AbstractMultimap.createCollection(AbstractMultimap.java:156)
    at com.google.common.collect.AbstractMultimap.getOrCreateCollection(AbstractMultimap.java:214)
    at com.google.common.collect.AbstractMultimap.put(AbstractMultimap.java:201)
    at com.google.common.collect.AbstractSetMultimap.put(AbstractSetMultimap.java:117)
    at com.google.common.collect.HashMultimap.put(HashMultimap.java:49)
    at com.google.common.eventbus.AnnotatedHandlerFinder.findAllHandlers(AnnotatedHandlerFinder.java:57)
    at com.google.common.eventbus.EventBus.register(EventBus.java:211)
    at org.apache.flume.node.Application.main(Application.java:355)

Answer 1

以下异常意味着水槽代理没有足够的内存（堆是特定的）来执行任务。

在flume_env.sh文件中增加flume代理的java内存，或在部署时使用flume ng代理-n NetcatAgent-f net.conf-Xmx2048m（注意：这会将flume堆大小设置为2GB=2048MB）

您可以从命令行指定-D和-xjava选项。