0

我正在尝试将 Hadoop 流与 python 脚本一起使用,但不幸的是我收到以下错误:

14/08/23 13:31:50 INFO streaming.StreamJob: To kill this job, run:
14/08/23 13:31:50 INFO streaming.StreamJob: UNDEF/bin/hadoop job  -Dmapred.job.tracker=localhost.localdomain:8021 -kill job_201408210627_0018
14/08/23 13:31:50 INFO streaming.StreamJob: Tracking URL: http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201408210627_0018
14/08/23 13:31:51 INFO streaming.StreamJob:  map 0%  reduce 0%
14/08/23 13:32:17 INFO streaming.StreamJob:  map 100%  reduce 100%
14/08/23 13:32:17 INFO streaming.StreamJob: To kill this job, run:
14/08/23 13:32:17 INFO streaming.StreamJob: UNDEF/bin/hadoop job  -Dmapred.job.tracker=localhost.localdomain:8021 -kill job_201408210627_0018
14/08/23 13:32:17 INFO streaming.StreamJob: Tracking URL: http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201408210627_0018
14/08/23 13:32:17 ERROR streaming.StreamJob: Job not successful. Error: NA
14/08/23 13:32:17 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

我正在尝试运行以下命令:

hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.7.0.jar -input "/user/cloudera/vecs" -output "/user/cloudera/vecs_output" -file /home/cloudera/vects/streaming/mapper.py -mapper mapper.py -file /home/cloudera/vects/streaming/reducer.py -reducer reducer.py -jobconf mapred.map.tasks=20 -jobconf mapred.reduce.tasks=1

当我查看工作设置时,我可以看到:

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

当我尝试使用命令在没有 hadoop 的情况下运行我的代码时,一切似乎都运行良好:

head -100 ./data/vecs.txt|./streaming/mapper.py|./streaming/reducer.py

我也通读了这篇文章,我有#!/usr/bin/env python2.7 作为我的python文件的第一行。

有谁知道可能出了什么问题?提前感谢您的任何建议和答案。

4

1 回答 1

0

因此,我通过从头开始使用 Hadoop 设置新的操作系统实例,使一切正常工作。

现在我只是好奇我的旧 Cloudera 虚拟机上可能存在什么问题?

于 2014-08-24T15:50:26.480 回答