1

我正在运行 RHadoop 的示例脚本来测试系统并使用以下命令。

library(rmr2)
library(rhdfs)
Sys.setenv(HADOOP_HOME="/usr/bin/hadoop")
Sys.setenv(HADOOP_CMD="/usr/bin/hadoop")
Sys.setenv(HADOOP_STREAMING="/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hadoop-mapreduce/hadoop-streaming.jar")
hdfs.init()
ints = to.dfs(1:100)
calc = mapreduce(input = ints, map = function(k, v) cbind(v, 2*v))

但它给了我一个像下面这样的错误。

>Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.streaming.AutoInputFormat not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1587)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1611)
13/08/21 18:30:25 INFO mapred.JobClient: Job complete: job_201308191923_0307
13/08/21 18:30:25 INFO mapred.JobClient: Counters: 7
13/08/21 18:30:25 INFO mapred.JobClient:   Job Counters
13/08/21 18:30:25 INFO mapred.JobClient:     Failed map tasks=1
13/08/21 18:30:25 INFO mapred.JobClient:     Launched map tasks=8
13/08/21 18:30:25 INFO mapred.JobClient:     Data-local map tasks=8
13/08/21 18:30:25 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=46647
13/08/21 18:30:25 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
13/08/21 18:30:25 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/08/21 18:30:25 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/08/21 18:30:25 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, in.folder = if (is.list(input)) { :
  hadoop streaming failed with error code 1

关于这里可能有什么问题的任何线索

4

1 回答 1

2

HADOOP_HOME 应该是一个目录。HADOOP_CMD 应该是一个程序。因此，由于它们被设置为相同的东西，那是错误的。但是 HADOOP_CMD 应该取代 HADOOP_HOME 所以这不应该是根本原因。所以剩下的唯一选择就是调试。如果您阅读了调试指南，您就会挖掘出 stderr 并且已经了解更多。仅使用控制台输出，没有什么可做的。

于 2013-08-23T03:47:45.507 回答