0

运行 mongo-hadoop 流时出现以下错误：

java.io.IOException: Cannot run program "mapper.py": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
    at java.lang.ProcessImpl.start(ProcessImpl.java:91)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 24 more

我通过 dumbo 或常规方式运行标准 Hadoop python 流没有问题。

在另一篇文章的 Hadoop python 流中提到了这个错误

我正在运行这样的工作：

hadoop jar /Volumes/Locodrive/hadoop/mongo-hadoop/streaming/target/mongo-hadoop-streaming-assembly-1.1.0-SNAPSHOT.jar -mapper mapper.py -file mapper.py -reducer reducer.py -file reducer.py -inputURI mongodb://localhost:27017/testdb.docs -outputURI mongodb://localhost:27017/testdb.testhadoop

在 mapper.py/reducer.py 上使用路径/绝对路径，在 -file 参数中添加绝对路径没有帮助。标准 Hadoop 流式传输没有任何问题，所以我没有收到错误。

添加mapper.py和reducer.py到 hdfs 也没有帮助。

mapper.py并且reducer.py是可执行的并且在第一行确实有一个shebang：

映射器.py

#!/usr/bin/env python

import sys
sys.path.append(".")


from pymongo_hadoop import BSONMapper


def mapper(documents):

    i = 0
    for doc in documents:
        i += 1
        yield {"_id": "test", "count": 1}


BSONMapper(mapper)
print >> sys.stderr, "Done Mapping!!!"

减速器.py

#!/usr/bin/env python
# encoding: utf-8

import sys
sys.path.append('.')


from pymongo_hadoop import BSONReducer


def reducer(key, values):
    print >> sys.stderr, "Processing key %s" % key
    _count = 0
    for v in values:
        _count += v["count"]

    return {"_id": key, "count": _count}

BSONReducer(reducer)

我在 OSX 上运行 cloudera Hadoop CDH3u3。Java 示例可以正常工作

更新

我尝试了 0.23.1 并得到了同样的错误。

running -debug 不会删除 PackagedJobJar streamjob.jar

当我提取它mapper.py并reducer.py在那里时

这些文件在运行 std 流作业时也存在。mongo-haddoop-streaming 仍然会产生上述错误

4

1 回答 1

0

k 让它工作它应该是 -files 而不是 -file

[http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/util/GenericOptionsParser.html][1]

hadoop jar /Volumes/Locodrive/hadoop/mongo-hadoop/streaming/target/mongo-hadoop-streaming-assembly-1.1.0-SNAPSHOT.jar -files mapper.py,reducer.py -inputURI mongodb://127.0.0.1:27017/mongo_hadoop.yield_historical.in -outputURI mongodb://127.0.0.1:27017/mongo_hadoop.testhadoop -mapper mapper.py -reducer reducer.py -verbose -debug
于 2012-06-27T18:16:01.913 回答