hadoop - Hadoop mapreduce 运行非常缓慢

Question

我正在使用 4datanode/1namenode hadoop 集群，版本 1.1.2 作为 vms 安装在 xenserver 中。我有一个 1GB 的文本文件并尝试进行字数统计。map 花了 2 小时，reducer 就挂了。一个普通的 perl 脚本在 10 分钟内完成了这项工作。看起来我的设置中缺少一些东西。

即使是 Kbs 中的小文件也需要很长时间。

hadoop@master ~]$ hadoop jar /usr/share/hadoop/hadoop-examples-1.1.2.jar  wordcount huge out
13/05/29 10:45:09 INFO input.FileInputFormat: Total input paths to process : 1
13/05/29 10:45:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/05/29 10:45:09 WARN snappy.LoadSnappy: Snappy native library not loaded
13/05/29 10:45:11 INFO mapred.JobClient: Running job: job_201305290801_0002
13/05/29 10:45:12 INFO mapred.JobClient:  map 0% reduce 0%
13/05/29 10:57:14 INFO mapred.JobClient:  map 2% reduce 0%
13/05/29 10:58:01 INFO mapred.JobClient:  map 3% reduce 0%
13/05/29 10:58:53 INFO mapred.JobClient:  map 4% reduce 0%
13/05/29 10:58:54 INFO mapred.JobClient:  map 5% reduce 0%
13/05/29 10:59:33 INFO mapred.JobClient:  map 6% reduce 0%
13/05/29 11:01:52 INFO mapred.JobClient:  map 7% reduce 0%
13/05/29 11:03:02 INFO mapred.JobClient:  map 8% reduce 0%
13/05/29 11:03:20 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_0, Status : FAILED
Task attempt_201305290801_0002_m_000002_0 failed to report status for 604 seconds. Killing!
13/05/29 11:03:28 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000003_0, Status : FAILED
Task attempt_201305290801_0002_m_000003_0 failed to report status for 604 seconds. Killing!
13/05/29 11:03:29 INFO mapred.JobClient:  map 9% reduce 0%
13/05/29 11:04:07 INFO mapred.JobClient:  map 10% reduce 0%
13/05/29 11:05:13 INFO mapred.JobClient:  map 11% reduce 0%
13/05/29 11:06:34 INFO mapred.JobClient:  map 12% reduce 0%
13/05/29 11:06:59 INFO mapred.JobClient:  map 13% reduce 0%
13/05/29 11:08:14 INFO mapred.JobClient:  map 14% reduce 0%
13/05/29 11:08:39 INFO mapred.JobClient:  map 15% reduce 0%
13/05/29 11:09:35 INFO mapred.JobClient:  map 16% reduce 0%
13/05/29 11:10:03 INFO mapred.JobClient:  map 17% reduce 0%
13/05/29 11:10:55 INFO mapred.JobClient:  map 18% reduce 0%
13/05/29 11:11:47 INFO mapred.JobClient:  map 19% reduce 0%
13/05/29 11:14:05 INFO mapred.JobClient:  map 20% reduce 0%
13/05/29 11:15:22 INFO mapred.JobClient:  map 21% reduce 0%
13/05/29 11:15:49 INFO mapred.JobClient:  map 22% reduce 0%
13/05/29 11:17:09 INFO mapred.JobClient:  map 23% reduce 0%
13/05/29 11:18:06 INFO mapred.JobClient:  map 24% reduce 0%
13/05/29 11:18:29 INFO mapred.JobClient:  map 25% reduce 0%
13/05/29 11:18:53 INFO mapred.JobClient:  map 26% reduce 0%
13/05/29 11:20:05 INFO mapred.JobClient:  map 27% reduce 0%
13/05/29 11:21:09 INFO mapred.JobClient:  map 28% reduce 0%
13/05/29 11:21:45 INFO mapred.JobClient:  map 29% reduce 0%
13/05/29 11:22:14 INFO mapred.JobClient:  map 30% reduce 0%
13/05/29 11:22:31 INFO mapred.JobClient:  map 31% reduce 0%
13/05/29 11:22:32 INFO mapred.JobClient:  map 32% reduce 0%
13/05/29 11:23:01 INFO mapred.JobClient:  map 33% reduce 0%
13/05/29 11:23:41 INFO mapred.JobClient:  map 34% reduce 0%
13/05/29 11:24:29 INFO mapred.JobClient:  map 35% reduce 0%
13/05/29 11:25:16 INFO mapred.JobClient:  map 36% reduce 0%
13/05/29 11:25:58 INFO mapred.JobClient:  map 37% reduce 0%
13/05/29 11:27:09 INFO mapred.JobClient:  map 38% reduce 0%
13/05/29 11:27:55 INFO mapred.JobClient:  map 39% reduce 0%
13/05/29 11:28:33 INFO mapred.JobClient:  map 40% reduce 0%
13/05/29 11:29:50 INFO mapred.JobClient:  map 41% reduce 0%
13/05/29 11:30:29 INFO mapred.JobClient:  map 42% reduce 0%
13/05/29 11:31:37 INFO mapred.JobClient:  map 43% reduce 0%
13/05/29 11:32:10 INFO mapred.JobClient:  map 44% reduce 0%
13/05/29 11:32:34 INFO mapred.JobClient:  map 45% reduce 0%
13/05/29 11:34:08 INFO mapred.JobClient:  map 46% reduce 0%
13/05/29 11:36:01 INFO mapred.JobClient:  map 47% reduce 0%
13/05/29 11:36:57 INFO mapred.JobClient:  map 48% reduce 0%
13/05/29 11:37:53 INFO mapred.JobClient:  map 49% reduce 0%
13/05/29 11:39:50 INFO mapred.JobClient:  map 50% reduce 0%
13/05/29 11:42:17 INFO mapred.JobClient:  map 51% reduce 0%
13/05/29 11:43:26 INFO mapred.JobClient:  map 52% reduce 0%
13/05/29 11:47:55 INFO mapred.JobClient:  map 53% reduce 0%
13/05/29 11:48:25 INFO mapred.JobClient:  map 54% reduce 0%
13/05/29 11:49:28 INFO mapred.JobClient:  map 54% reduce 2%
13/05/29 11:49:31 INFO mapred.JobClient:  map 54% reduce 4%
13/05/29 11:50:03 INFO mapred.JobClient:  map 55% reduce 4%
13/05/29 11:50:49 INFO mapred.JobClient:  map 56% reduce 4%
13/05/29 11:50:54 INFO mapred.JobClient:  map 58% reduce 4%
13/05/29 11:51:21 INFO mapred.JobClient:  map 59% reduce 4%
13/05/29 11:51:46 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_1, Status : FAILED
Task attempt_201305290801_0002_m_000002_1 failed to report status for 685 seconds. Killing!
13/05/29 11:52:09 INFO mapred.JobClient:  map 61% reduce 4%
13/05/29 11:52:27 INFO mapred.JobClient:  map 62% reduce 4%
13/05/29 11:52:53 INFO mapred.JobClient:  map 63% reduce 4%
13/05/29 11:53:36 INFO mapred.JobClient:  map 64% reduce 4%
13/05/29 11:53:57 INFO mapred.JobClient:  map 65% reduce 4%
13/05/29 11:54:41 INFO mapred.JobClient:  map 66% reduce 4%
13/05/29 11:55:51 INFO mapred.JobClient:  map 67% reduce 4%
13/05/29 11:57:00 INFO mapred.JobClient:  map 68% reduce 4%
13/05/29 11:57:04 INFO mapred.JobClient:  map 69% reduce 4%
13/05/29 11:57:11 INFO mapred.JobClient:  map 70% reduce 4%
13/05/29 11:57:41 INFO mapred.JobClient:  map 71% reduce 4%
13/05/29 11:58:13 INFO mapred.JobClient:  map 72% reduce 4%
13/05/29 11:58:45 INFO mapred.JobClient:  map 73% reduce 4%
13/05/29 11:59:05 INFO mapred.JobClient:  map 74% reduce 4%
13/05/29 11:59:08 INFO mapred.JobClient:  map 74% reduce 6%
13/05/29 11:59:42 INFO mapred.JobClient:  map 75% reduce 6%
13/05/29 11:59:52 INFO mapred.JobClient:  map 76% reduce 6%
13/05/29 12:00:33 INFO mapred.JobClient:  map 77% reduce 6%
13/05/29 12:00:53 INFO mapred.JobClient:  map 78% reduce 6%
13/05/29 12:01:06 INFO mapred.JobClient:  map 79% reduce 6%
13/05/29 12:01:51 INFO mapred.JobClient:  map 80% reduce 6%
13/05/29 12:02:29 INFO mapred.JobClient:  map 81% reduce 6%
13/05/29 12:02:39 INFO mapred.JobClient:  map 82% reduce 6%
13/05/29 12:02:56 INFO mapred.JobClient:  map 83% reduce 6%
13/05/29 12:03:36 INFO mapred.JobClient:  map 84% reduce 6%
13/05/29 12:04:05 INFO mapred.JobClient:  map 85% reduce 6%
13/05/29 12:04:59 INFO mapred.JobClient:  map 86% reduce 6%
13/05/29 12:05:47 INFO mapred.JobClient:  map 87% reduce 6%
13/05/29 12:07:04 INFO mapred.JobClient:  map 88% reduce 6%
13/05/29 12:08:00 INFO mapred.JobClient:  map 89% reduce 6%
13/05/29 12:08:32 INFO mapred.JobClient:  map 90% reduce 6%
13/05/29 12:09:41 INFO mapred.JobClient:  map 91% reduce 6%
13/05/29 12:10:04 INFO mapred.JobClient:  map 92% reduce 6%
13/05/29 12:10:17 INFO mapred.JobClient:  map 93% reduce 6%
13/05/29 12:10:45 INFO mapred.JobClient:  map 94% reduce 6%
13/05/29 12:10:49 INFO mapred.JobClient:  map 95% reduce 6%
13/05/29 12:11:00 INFO mapred.JobClient:  map 96% reduce 6%
13/05/29 12:11:03 INFO mapred.JobClient:  map 97% reduce 6%
13/05/29 12:11:12 INFO mapred.JobClient:  map 98% reduce 6%
13/05/29 12:11:17 INFO mapred.JobClient:  map 99% reduce 6%
13/05/29 12:12:02 INFO mapred.JobClient:  map 100% reduce 6%






^C[hadoop@master ~]$

score 1 · Accepted Answer

从您提供的有限信息（控制台输出）来看，集群看起来并不健康。

13/05/29 11:03:20 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_0, Status : FAILED
Task attempt_201305290801_0002_m_000002_0 failed to report status for 604 seconds. Killing!

在 10 分钟内未向 JobTracker 报告的某个节点上尝试了任务。这导致任务再次被重新安排。深入研究更多日志，确定哪些特定节点未能完成分配的任务可能是您应该做的事情。

hadoop - Hadoop mapreduce 运行非常缓慢

1 回答 1

Related

Reference