0

在我的 map-reduce 作业中,当我将 reducer 的数量设置为 0 时,程序在不到 5 分钟的时间内成功运行到最后。但是,当我将 reducer 的数量设置为某个值而不是 0(1-40 之间)甚至没有指定 reducer 类时,它在执行许多失败的任务时会非常缓慢,并且突然失败并显示以下错误消息。由于我没有指定减速器类,唯一可能导致此错误的是对映射器的输出进行排序。映射器产生大约 27,000 个不同的键。这里有什么问题?日志文件中也没有任何东西可以提供帮助。我唯一的猜想是大量的键导致了这个问题。我怎么解决这个问题?

  13/10/28 13:06:40 INFO mapred.JobClient:  map 70% reduce 18%
  13/10/28 13:07:13 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000052_2, Status : FAILED
  Task attempt_201310101346_0661_m_000052_2 failed to report status for 600 seconds. Killing!
  13/10/28 13:07:19 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000059_2, Status : FAILED
  Task attempt_201310101346_0661_m_000059_2 failed to report status for 600 seconds. Killing!
  13/10/28 13:07:23 INFO mapred.JobClient:  map 70% reduce 19%
  13/10/28 13:07:24 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000060_2, Status : FAILED
 Task attempt_201310101346_0661_m_000060_2 failed to report status for 600 seconds. Killing!
 13/10/28 13:07:26 INFO mapred.JobClient:  map 70% reduce 20%
 13/10/28 13:07:47 INFO mapred.JobClient:  map 71% reduce 20%
 13/10/28 13:07:54 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000061_2, Status : FAILED
 Task attempt_201310101346_0661_m_000061_2 failed to report status for 600 seconds. Killing!
 13/10/28 13:08:22 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000062_2, Status : FAILED
 Task attempt_201310101346_0661_m_000062_2 failed to report status for 600 seconds. Killing!
 13/10/28 13:09:03 INFO mapred.JobClient:  map 72% reduce 20%
 13/10/28 13:09:15 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000063_2, Status : FAILED
 Task attempt_201310101346_0661_m_000063_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:18 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000066_2, Status : FAILED
 Task attempt_201310101346_0661_m_000066_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:20 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000068_2,     Status : FAILED
Task attempt_201310101346_0661_m_000068_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:55 INFO mapred.JobClient:  map 72% reduce 21%
13/10/28 13:10:30 INFO mapred.JobClient:  map 72% reduce 22%
13/10/28 13:10:33 INFO mapred.JobClient:  map 73% reduce 22%
13/10/28 13:12:21 INFO mapred.JobClient:  map 74% reduce 22%
13/10/28 13:13:40 INFO mapred.JobClient:  map 74% reduce 23%
13/10/28 13:14:09 INFO mapred.JobClient: Job complete: job_201310101346_0661
13/10/28 13:14:09 INFO mapred.JobClient: Counters: 31
13/10/28 13:14:09 INFO mapred.JobClient:   File System Counters
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of bytes read=0
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of bytes written=542433323
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of read operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     FILE: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of bytes read=316452240
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of bytes written=0
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of read operations=100
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient:     HDFS: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient:   Job Counters 
13/10/28 13:14:09 INFO mapred.JobClient:     Failed map tasks=1
13/10/28 13:14:09 INFO mapred.JobClient:     Launched map tasks=169
13/10/28 13:14:09 INFO mapred.JobClient:     Launched reduce tasks=1
13/10/28 13:14:09 INFO mapred.JobClient:     Data-local map tasks=23
13/10/28 13:14:09 INFO mapred.JobClient:     Rack-local map tasks=146
13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=72419457
13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=1194624
13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
 13/10/28 13:14:09 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
 13/10/28 13:14:09 INFO mapred.JobClient:   Map-Reduce Framework
13/10/28 13:14:09 INFO mapred.JobClient:     Map input records=6894035
13/10/28 13:14:09 INFO mapred.JobClient:     Map output records=6422195
 13/10/28 13:14:09 INFO mapred.JobClient:     Map output bytes=2300749768
13/10/28 13:14:09 INFO mapred.JobClient:     Input split bytes=6200
13/10/28 13:14:09 INFO mapred.JobClient:     Combine input records=6422195
13/10/28 13:14:09 INFO mapred.JobClient:     Combine output records=801608
13/10/28 13:14:09 INFO mapred.JobClient:     Spilled Records=801608
13/10/28 13:14:09 INFO mapred.JobClient:     CPU time spent (ms)=48077980
13/10/28 13:14:09 INFO mapred.JobClient:     Physical memory (bytes) snapshot=13246660608
13/10/28 13:14:09 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=20080971776
 13/10/28 13:14:09 INFO mapred.JobClient:     Total committed heap usage (bytes)=10181214208
 13/10/28 13:14:09 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
 13/10/28 13:14:09 INFO mapred.JobClient:     BYTES_READ=316416597
 13/10/28 13:14:09 INFO mapred.JobClient: Job Failed: NA
 Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1373)
    at Mondrian8.main(Mondrian8.java:547)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
4

0 回答 0