使用apache水槽将数据流式传输到hbase

Question

我正在尝试使用apache flume将数据加载到hbase中。当我使用flume将数据传输到hadoop时，它工作得很好。但是当我启动flume代理将数据加载到hbase时，我得到了NoClassDefFoundError。

14/05/12 23:14:10 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:agent4.conf
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Added sinks: sink1 Agent: agent4
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Processing:sink1
14/05/12 23:14:10 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent4]
14/05/12 23:14:10 INFO node.AbstractConfigurationProvider: Creating channels
14/05/12 23:14:10 INFO channel.DefaultChannelFactory: Creating instance of channel channel1 type FILE
14/05/12 23:14:10 INFO node.AbstractConfigurationProvider: Created channel channel1
14/05/12 23:14:10 INFO source.DefaultSourceFactory: Creating instance of source source1, type exec
14/05/12 23:14:10 INFO sink.DefaultSinkFactory: Creating instance of sink: sink1, type: org.apache.flume.sink.hbase.HBaseSink
14/05/12 23:14:10 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
    at org.apache.flume.sink.hbase.HBaseSink.<init>(HBaseSink.java:102)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at java.lang.Class.newInstance(Class.java:374)
    at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:43)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 17 more

这是我的水槽配置:

flume-env.sh

JAVA_HOME=/usr
FLUME_CLASSPATH=/home/alpha/apache-flume-1.4.0-bin/lib
HBASE_CLASSPATH=/home/alpha/hbase-0.98.1/lib
HBASE_HOME=/home/alpha/hbase-0.98.1
FLUME_HOME=/home/alpha/apache-flume-1.4.0-bin

代理4.conf

# Name the components on this agent
agent4.sources = source1
agent4.sinks = sink1
agent4.channels = channel1

# Describe/configure source1
agent4.sources.source1.type = exec
agent4.sources.source1.command = tail -f /tmp/testGenerate.csv

# Describe sink1
agent4.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
agent4.sinks.sink1.table = AdreamLumiHB
agent4.sinks.sink1.columnFamily =lumiCF
agent4.sinks.sink1.batchSize=5000
agent4.sinks.sink1.serializer.regex = ^(\d+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),.*
agent4.sinks.sink1.serializer.regexIgnoreCase = true
agent4.sinks.sink1.serializer.colNames = id,nom,valeur,batiment,etage,piece

# Use a channel which buffers events to a file
agent4.channels.channel1.type = FILE 
agent4.channels.channel1.transactionCapacity = 1000000 
agent4.channels.channel1.checkpointInterval 30000
agent4.channels.channel1.maxFileSize = 2146435071
agent4.channels.channel1.capacity 10000000 

# Bind the source and sink to the channel
agent4.sources.source1.channels = channel1
agent4.sinks.sink1.channel = channel1

Answer 1

我建议将hbase home的/lib文件夹中的所有jar复制到flume的/lib文件夹中。这帮助我解决了这个问题。

Answer 2

当您将hbase类路径附加到水槽时会发生什么？

FLUME_CLASSPATH=/home/alpha/apache-flume-1.4.0-bin/lib/\*:/home/alpha/hbase-0.98.1/lib/\*

注意：星号*前不要包括反斜杠。我把它放在那里是因为星号不会出现在这个编辑器上。