hadoop使用mongo-hadoop流到python

Question

我试图得到地图减少功能与Python使用mongo-hadoop。hadoop正在工作，hadoop流媒体正在与python一起工作，mongo-hadoop适配器正在工作。然而，使用python的mongo-hadoop流媒体示例不起作用。当试图在流/示例/金库中运行示例时，我得到以下错误：

$user@host：~/git/mongo-hadoop/streaming$hadoop-jar-target/mongo-hadoop-streaming-assembly-1.0。1.jar-mapper-examples/treasury/mapper。py-减速器示例/库存/减速器。py-inputformatcom.mongodb.hadoop。映射。MongoInputFormat-outputformat com.mongodb.hadoop。映射。MongoOutputFormat-inputURImongodb://127.0.0.1/mongo_hadoop.yield_historical.in-输出URImongodb://127.0.0.1/mongo_hadoop.yield_historical.streaming.out

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Running

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Init

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Process Args

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Setup Options'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: PreProcess Args

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Parse Options

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-mapper'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'examples/treasury/mapper.py'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-reducer'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'examples/treasury/reducer.py'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-inputformat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'com.mongodb.hadoop.mapred.MongoInputFormat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-outputformat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'com.mongodb.hadoop.mapred.MongoOutputFormat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-inputURI'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'mongodb://127.0.0.1/mongo_hadoop.yield_historical.in'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-outputURI'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'mongodb://127.0.0.1/mongo_hadoop.yield_historical.streaming.out'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Add InputSpecs

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Setup output_

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Post Process Args

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Args processed.

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

**Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/filecache/DistributedCache**
    at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:959)
    at com.mongodb.hadoop.streaming.MongoStreamJob.run(MongoStreamJob.java:36)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.mongodb.hadoop.streaming.MongoStreamJob.main(MongoStreamJob.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.filecache.DistributedCache
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 10 more

如果有人能透露一些信息，那将是一个很大的帮助。

完整信息：

据我所知，我需要让以下四件事发挥作用：

1. 安装并测试hadoop
2. 使用python安装和测试hadoop流
3. 安装并测试mongo-hadoop
4. 使用python安装和测试mongo-hadoop流

简而言之，我已经把一切都做好了，直到第四步。使用（https://github.com/danielpoe/cloudera）我已经安装了云4

1. 使用主厨食谱云4已安装，并正在运行和测试
2. 使用michael nolls博客教程，用python测试hadoop流媒体成功
3. 使用mongodb.org的文档能够运行财政部和ufo示例（build.sbt构建cdh4）
4. 使用twitter示例的自述文件下载了1.5小时的twitter数据，并尝试了财政部示例。

Answer 1

你安装了最新的pymongo_hadoop连接器吗？您运行的其他软件的版本是什么？