1

我为 hadoop 流运行以下命令

hadoop jar contrib/streaming/hadoop-streaming-1.0.3.jar -input test1/testregxml.xml -output testoutput2 -file ~/Programs/CMapper.php -mapper ~/Programs/CMapper.php -jobconf mapred.reduce.tasks=1

hadoop 任务失败，输出如下

12/10/26 15:36:56 WARN streaming.StreamJob: -jobconf option is deprecated, please use -

D instead.
packageJobJar: [/home/HadoopNaman1/Programs/CMapper.php, /tmp/hadoop-HadoopNaman1/hadoop-unjar8119734358986860064/] [] /tmp/streamjob3840254105417044886.jar tmpDir=null
12/10/26 15:36:58 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/10/26 15:36:58 WARN snappy.LoadSnappy: Snappy native library not loaded
12/10/26 15:36:58 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/26 15:36:58 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-HadoopNaman1/mapred/local]
12/10/26 15:36:58 INFO streaming.StreamJob: Running job: job_201209282004_0015
12/10/26 15:36:58 INFO streaming.StreamJob: To kill this job, run:
12/10/26 15:36:58 INFO streaming.StreamJob: /home/HadoopNaman1/hadoop-1.0.3/libexec/../bin/hadoop job  -Dmapred.job.tracker=saraswati.tallysolutions.com:9051 -kill job_201209282004_0015
12/10/26 15:36:58 INFO streaming.StreamJob: Tracking URL: http://192.168.1.17:50030/jobdetails.jsp?jobid=job_201209282004_0015
12/10/26 15:36:59 INFO streaming.StreamJob:  map 0%  reduce 0%
12/10/26 15:37:51 INFO streaming.StreamJob:  map 100%  reduce 100%
12/10/26 15:37:51 INFO streaming.StreamJob: To kill this job, run:
12/10/26 15:37:51 INFO streaming.StreamJob: /home/HadoopNaman1/hadoop-1.0.3/libexec/../bin/hadoop job  -Dmapred.job.tracker=saraswati.tallysolutions.com:9051 -kill job_201209282004_0015
12/10/26 15:37:51 INFO streaming.StreamJob: Tracking URL: http://192.168.1.17:50030/jobdetails.jsp?jobid=job_201209282004_0015
12/10/26 15:37:51 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201209282004_0015_m_000001
12/10/26 15:37:51 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

这是我的jobtracker日志

2012-10-26 15:37:44,906 INFO org.apache.hadoop.mapred.TaskInProgress: TaskInProgress task_201209282004_0015_m_000001 has failed 4 times.
2012-10-26 15:37:44,906 INFO org.apache.hadoop.mapred.JobInProgress: Aborting job job_201209282004_0015
2012-10-26 15:37:44,906 INFO org.apache.hadoop.mapred.JobInProgress: Killing job 'job_201209282004_0015'
2012-10-26 15:37:44,907 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201209282004_0015_m_000002_0' to tip task_201209282004_0015_m_000002, for tracker 'tracker_192.168.1.86:localhost/127.0.0.1:45880'
2012-10-26 15:37:44,907 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000001_3'
2012-10-26 15:37:46,861 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201209282004_0015_m_000000_3: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
        ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
        at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230)
        at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
        ... 22 more
Caused by: java.io.IOException: Cannot run program "/tmp/hadoop-HadoopNaman1/mapred/local/taskTracker/HadoopNaman1/jobcache/job_201209282004_0015/attempt_201209282004_0015_m_000000_3/work/./CMapper.php": java.io.IOException: error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
        at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
        ... 23 more
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
        at java.lang.ProcessImpl.start(ProcessImpl.java:81)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
        ... 24 more

2012-10-26 15:37:49,866 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000000_3'
2012-10-26 15:37:50,913 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201209282004_0015_m_000002_0' has completed task_201209282004_0015_m_000002 successfully.
2012-10-26 15:37:50,914 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201209282004_0015,submitTime=1351246018628,launchTime=1351246018806,firstMapTaskLaunchTime=1351246028873,firstJobSetupTaskLaunchTime=1351246019863,firstJobCleanupTaskLaunchTime=1351246064906,finishTime=1351246070914,numMaps=2,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=HadoopNaman1,queue=default,status=FAILED,mapSlotSeconds=60,reduceSlotsSeconds=0,clusterMapCapacity=12,clusterReduceCapacity=12,jobName=streamjob3840254105417044886.jar
2012-10-26 15:37:50,936 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/job_201209282004_0015_1351246018628_HadoopNaman1_streamjob3840254105417044886.jar to file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/done/version-1/saraswati.tallysolutions.com_1348842869935_/2012/10/26/000000
2012-10-26 15:37:50,942 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/job_201209282004_0015_conf.xml to file:/home/HadoopNaman1/hadoop-1.0.3/logs/history/done/version-1/saraswati.tallysolutions.com_1348842869935_/2012/10/26/000000
2012-10-26 15:37:50,948 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000002_0'
2012-10-26 15:37:50,948 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201209282004_0015_m_000003_0'
2012-10-26 15:37:51,850 INFO org.apache.hadoop.mapred.JobTracker: Killing job job_201209282004_0015

请提出一些建议。映射器文件是可执行的，并且存在于权限设置为 777 的目录中。

4

1 回答 1

1

可能是在抱怨您的解释器（首先在文件中，例如“#!/usr/bin/php”）指向该路径中不存在的解释器。当事情在这个基础层面被破坏时，Hadoop 会给出一些非常奇怪/难以破译的错误。

于 2012-12-02T04:23:48.157 回答