HDFS数据上的配置单元查询异常

Question

我在执行配置单元查询时遇到异常。我关注以下链接：http://www.thecloudavenue.com/2013/03/analysis-tweets-using-flume-hadoop-and.html

终端数据在这里：

        hive> select id from tweets;
        Total jobs = 1
        Launching Job 1 out of 1
        Number of reduce tasks is set to 0 since there's no reduce operator
        Starting Job = job_201502231939_0005, Tracking URL =      
        http://localhost:50030/jobdetails.jsp?jobid=job_201502231939_0005
        Kill Command = /prj/hadoop/libexec/../bin/hadoop job  -kill    
        job_201502231939_0005
       Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
       2015-02-23 20:00:05,683 Stage-1 map = 0%,  reduce = 0%
       2015-02-23 20:00:27,016 Stage-1 map = 100%,  reduce = 100%
       Ended Job = job_201502231939_0005 with errors
       Error during job, obtaining debugging information...
       Job Tracking URL: http://localhost:50030/jobdetails.jsp?   jobid=job_201502231939_0005
       Examining task ID: task_201502231939_0005_m_000002 (and more) from job job_201502231939_0005

       Task with the most failures(4): 
       -----
       Task ID:
         task_201502231939_0005_m_000000

       URL:
         http://localhost:50030/taskdetails.jsp?   jobid=job_201502231939_0005&tipid=task_201502231939_0005_m_000000
    -----
       Diagnostic Messages for this Task:
       java.lang.RuntimeException: Error in configuring object
        at    
       org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
           at    org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at    org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
           at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
           at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 9 more
    Caused by: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
        ... 14 more
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 17 more
    Caused by: java.lang.RuntimeException: Map operator initialization failed
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)
        ... 22 more
    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe
        at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:335)
        at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:353)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:123)
        ... 22 more
    Caused by: java.lang.ClassNotFoundException: com.cloudera.hive.serde.JSONSerDe
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:274)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
        at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:305)
        ... 24 more


    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
    MapReduce Jobs Launched: 
    Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
    Total MapReduce CPU Time Spent: 0 msec

Answer 1

将flume-sources-1.0-snapshot.jar复制到/usr/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-snapshot.jar，如果未使用默认值，还可以复制到/var/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-snapshot.jar或它们各自的路径。

还可以使用以下命令在配置单元shell中添加jar：

ADD JAR /local/path/to/the/jar