0

I have installed HDP 2.x, and hue. (2 hosts)

I need to solve this issue as I'm new to oozie workflows. I followed example here: http://www.oraclealchemist.com/news/hadoop-streaming-hue-oozie-workflows-hive/ It works perfectly if I have mapper and reducers in all hosts at respective paths only. I see ambari-qa smoke test of oozie is fine but when I submit a streaming job from Hue UI for Oozie, where mapper and reducers where shell scripts which performs word count (term frequency) and I wanted to use "Add File" (-file in hadoop command line) section in Oozie workflow editor and upload mapper and reducers to HDFS. After this change and submitting this job, there was an error. I get the following error for all attempts of map tasks:

2013-12-16 19:21:24,278 ERROR [main] org.apache.hadoop.streaming.PipeMapRed: configuration exception java.io.IOException: Cannot run program "/hadoop/yarn/local/usercache/root/appcache/application_1387201627160_0006/container_1387201627160_0006_01_000002/./maptf.sh": java.io.IOException: error=2, No such file or directory

This means it can't able to find mapper and reducer in that path where oozie/mapred/yarn will create files onfly. Do I have any oozie configuration and workflow issues? (logs by email: sandeepboda91083@gmail.com)

In HDFS, I have all paths and files setup correctly under root user.

Note: I can able to run streaming jobs without oozie as:

cd /root/mrtest/
ls
-rwxrwxrwx 1 root root 235 Dec 11 11:37 maptf.sh
-rwxrwxrwx 1 root root 273 Dec 11 11:37 redtf.sh

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.2.0.2.0.6.0-76.jar -D stream.num.map.output.key.fields=1 -input crane_in1 -output crane_out2 -file ./maptf.sh -mapper maptf.sh -file ./redtf.sh -reducer redtf.sh

It seems I cant attach logs here. Please mail me for files and logs.

4

1 回答 1

1

这可能是与Hue 列表中的这些用户相同的问题。

如果您使用文件浏览器编辑文件,它们可能会因为一些非 Unix 新行而损坏。您应该直接上传它们而不是修改它们或使用更新版本的Hue(在撰写本文时为 3.5)。

于 2013-12-18T14:30:06.413 回答