5

我试图运行一个 Ruby Hadoop 流程序,它在“Ruby 权威指南”中给出。

这是我使用的命令:

hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2+737.jar 
-input input/temperature -output output -mapper /home/cloudera/projects/max_temp/map.rb 
-reducer /home/cloudera/projects/max_temp/reduce.rb

文件路径是正确的。运行命令后,我会收到如下错误:

packageJobJar: [/var/lib/hadoop-0.20/cache/cloudera/hadoop-unjar7505856816337735719/] [] /tmp/streamjob5440613434295562761.jar tmpDir=null
11/03/14 22:58:56 INFO mapred.FileInputFormat: Total input paths to process : 1
11/03/14 22:58:56 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop-0.20/cache/cloudera/mapred/local]
11/03/14 22:58:56 INFO streaming.StreamJob: Running job: job_201103140235_0015
11/03/14 22:58:56 INFO streaming.StreamJob: To kill this job, run:
11/03/14 22:58:56 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201103140235_0015
11/03/14 22:58:56 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp...
11/03/14 22:58:57 INFO streaming.StreamJob: map 0% reduce 0%
11/03/14 22:59:54 INFO streaming.StreamJob: map 100% reduce 100%
11/03/14 22:59:54 INFO streaming.StreamJob: To kill this job, run:
11/03/14 22:59:54 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201103140235_0015
11/03/14 22:59:54 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp...
11/03/14 22:59:54 ERROR streaming.StreamJob: Job not Successful!
11/03/14 22:59:54 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

我查看了地图日志文件,这是错误:

java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:379)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: configuration exception
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
... 22 more
Caused by: java.io.IOException: Cannot run program "/home/cloudera/projects/max_temp/map.rb": java.io.IOException: error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 23 more
Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 24 more 
4

2 回答 2

4

感谢 Lev Khomich 的回复。我已通过在命令中提及文件参数来解决此问题:

hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2+737.jar \
-input input/temperature -output output -mapper /home/cloudera/projects/max_temp/map.rb \
-reducer /home/cloudera/projects/max_temp/reduce.rb \
-file /home/cloudera/projects/words/map.rb \
-file /home/cloudera/projects/words/reduce.rb
于 2011-03-17T04:35:09.590 回答
0

我和你有同样的错误,并通过将'map.rb'和'reduce.rb'文件复制到/tmp/来解决问题,然后使用parms作为-mapper /tmp/map.rb -reducer /tmp/reduce.rb

于 2013-05-09T16:07:40.297 回答