我正在尝试使用 mongo-hadoop 使用 python 获得 map-reduce 功能。Hadoop 正在工作,hadoop 流正在与 python 一起工作,并且 mongo-hadoop 适配器正在工作。但是,使用 python 的 mongo-hadoop 流式传输示例不起作用。尝试在流/示例/金库中运行示例时,出现以下错误:
$user@host: ~/git/mongo-hadoop/streaming$ hadoop jar 目标/mongo-hadoop-streaming-assembly-1.0.1.jar -mapper 示例/treasury/mapper.py -reducer 示例/treasury/reducer.py -inputformat com.mongodb.hadoop.mapred.MongoInputFormat -outputformat com.mongodb.hadoop.mapred.MongoOutputFormat -inputURI mongodb://127.0.0.1/mongo_hadoop.yield_historical.in -outputURI mongodb://127.0.0.1/mongo_hadoop.yield_historical .streaming.out
13/04/09 11:54:34 INFO streaming.MongoStreamJob: Running
13/04/09 11:54:34 INFO streaming.MongoStreamJob: Init
13/04/09 11:54:34 INFO streaming.MongoStreamJob: Process Args
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Setup Options'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: PreProcess Args
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Parse Options
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-mapper'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'examples/treasury/mapper.py'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-reducer'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'examples/treasury/reducer.py'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-inputformat'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'com.mongodb.hadoop.mapred.MongoInputFormat'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-outputformat'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'com.mongodb.hadoop.mapred.MongoOutputFormat'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-inputURI'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'mongodb://127.0.0.1/mongo_hadoop.yield_historical.in'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-outputURI'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'mongodb://127.0.0.1/mongo_hadoop.yield_historical.streaming.out'
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Add InputSpecs
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Setup output_
13/04/09 11:54:34 INFO streaming.StreamJobPatch: Post Process Args
13/04/09 11:54:34 INFO streaming.MongoStreamJob: Args processed.
13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson
13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson
13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson
13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson
**Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/filecache/DistributedCache**
at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:959)
at com.mongodb.hadoop.streaming.MongoStreamJob.run(MongoStreamJob.java:36)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.mongodb.hadoop.streaming.MongoStreamJob.main(MongoStreamJob.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.filecache.DistributedCache
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 10 more
如果有人能提供一些启示,那将是一个很大的帮助。
完整信息:
据我所知,我需要完成以下四件事:
- 安装和测试hadoop
- 使用 python 安装和测试 hadoop 流
- 安装和测试 mongo-hadoop
- 使用 python 安装和测试 mongo-hadoop 流
因此,短处是我已经完成了第四步的所有工作。使用(https://github.com/danielpoe/cloudera)我已经安装了cloudera 4
- 使用 chef recipe cloudera 4 已安装并已启动并运行和测试
- 使用 michael nolls 博客教程,用 python 测试 hadoop 流式成功
- 使用 mongodb.org 上的文档能够运行库和 ufo 示例(在 build.sbt 中构建 cdh4)
- 使用流/示例中的 twitter 示例的自述文件下载了 1.5 小时的 twitter 数据,并尝试了财政部示例。