7

我正在尝试使用 mongo-hadoop 使用 python 获得 map-reduce 功能。Hadoop 正在工作,hadoop 流正在与 python 一起工作,并且 mongo-hadoop 适配器正在工作。但是,使用 python 的 mongo-hadoop 流式传输示例不起作用。尝试在流/示例/金库中运行示例时,出现以下错误:

$user@host: ~/git/mongo-hadoop/streaming$ hadoop jar 目标/mongo-hadoop-streaming-assembly-1.0.1.jar -mapper 示例/treasury/mapper.py -reducer 示例/treasury/reducer.py -inputformat com.mongodb.hadoop.mapred.MongoInputFormat -outputformat com.mongodb.hadoop.mapred.MongoOutputFormat -inputURI mongodb://127.0.0.1/mongo_hadoop.yield_historical.in -outputURI mongodb://127.0.0.1/mongo_hadoop.yield_historical .streaming.out

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Running

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Init

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Process Args

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Setup Options'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: PreProcess Args

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Parse Options

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-mapper'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'examples/treasury/mapper.py'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-reducer'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'examples/treasury/reducer.py'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-inputformat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'com.mongodb.hadoop.mapred.MongoInputFormat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-outputformat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'com.mongodb.hadoop.mapred.MongoOutputFormat'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-inputURI'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'mongodb://127.0.0.1/mongo_hadoop.yield_historical.in'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: '-outputURI'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Arg: 'mongodb://127.0.0.1/mongo_hadoop.yield_historical.streaming.out'

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Add InputSpecs

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Setup output_

13/04/09 11:54:34 INFO streaming.StreamJobPatch: Post Process Args

13/04/09 11:54:34 INFO streaming.MongoStreamJob: Args processed.

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

13/04/09 11:54:36 INFO io.MongoIdentifierResolver: Resolving: bson

**Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/filecache/DistributedCache**
    at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:959)
    at com.mongodb.hadoop.streaming.MongoStreamJob.run(MongoStreamJob.java:36)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.mongodb.hadoop.streaming.MongoStreamJob.main(MongoStreamJob.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.filecache.DistributedCache
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 10 more

如果有人能提供一些启示,那将是一个很大的帮助。

完整信息:

据我所知,我需要完成以下四件事:

  1. 安装和测试hadoop
  2. 使用 python 安装和测试 hadoop 流
  3. 安装和测试 mongo-hadoop
  4. 使用 python 安装和测试 mongo-hadoop 流

因此,短处是我已经完成了第四步的所有工作。使用(https://github.com/danielpoe/cloudera)我已经安装了cloudera 4

  1. 使用 chef recipe cloudera 4 已安装并已启动并运行和测试
  2. 使用 michael nolls 博客教程,用 python 测试 hadoop 流式成功
  3. 使用 mongodb.org 上的文档能够运行库和 ufo 示例(在 build.sbt 中构建 cdh4)
  4. 使用流/示例中的 twitter 示例的自述文件下载了 1.5 小时的 twitter 数据,并尝试了财政部示例。
4

1 回答 1

0

您是否安装了最新的 pymongo_hadoop 连接器?您正在运行什么版本的其他软件?

于 2013-04-15T13:50:39.853 回答