3

我已经使用以下站点的 manul 将 hadoop 安装并配置为单节点。

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#running-a-mapreduce-job

我已经编译了 wordcount 示例并运行它，但它需要很长时间并生成Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.

hduser@aptunix0043:/usr/local/hadoop/src$ hadoop jar WordCount.jar org/apache/hadoop/examples/WordCount input  ot

****hdfs://localhost:54310/user/hduser/input
12/07/03 02:52:35 INFO input.FileInputFormat: Total input paths to process : 1
12/07/03 02:52:36 INFO mapred.JobClient: Running job: job_201207030248_0002
12/07/03 02:52:37 INFO mapred.JobClient:  map 0% reduce 0%
12/07/03 02:52:52 INFO mapred.JobClient:  map 100% reduce 0%
12/07/03 03:21:26 INFO mapred.JobClient: Task Id :attempt_201207030248_0002_r_000000_0, Status : FAILED 
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.

12/07/03 03:21:47 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 03:22:08 WARN mapred.JobClient: Error reading task outputConnection timed out
 /user/hduser/input/*12/07/03 03:50:01 INFO mapred.JobClient: Task Id :      attempt_201207030248_0002_r_000000_1, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
12/07/03 03:50:22 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 03:50:43 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 04:18:35 INFO mapred.JobClient: Task Id :  attempt_201207030248_0002_r_000000_2, Status : FAILED
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
12/07/03 04:18:56 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 04:19:17 WARN mapred.JobClient: Error reading task outputConnection timed out
12/07/03 04:47:15 INFO mapred.JobClient: Job complete: job_201207030248_0002
12/07/03 04:47:15 INFO mapred.JobClient: Counters: 23
12/07/03 04:47:15 INFO mapred.JobClient:   Job Counters
12/07/03 04:47:15 INFO mapred.JobClient:     Launched reduce tasks=4
12/07/03 04:47:15 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=12948
12/07/03 04:47:15 INFO mapred.JobClient:     Total time spent by all reduces waiting  after reserving slots (ms)=0
12/07/03 04:47:15 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/07/03 04:47:15 INFO mapred.JobClient:     Launched map tasks=1
12/07/03 04:47:15 INFO mapred.JobClient:     Data-local map tasks=1
12/07/03 04:47:15 INFO mapred.JobClient:     Failed reduce tasks=1
12/07/03 04:47:15 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=16469
12/07/03 04:47:15 INFO mapred.JobClient:   FileSystemCounters
12/07/03 04:47:15 INFO mapred.JobClient:     HDFS_BYTES_READ=661744
12/07/03 04:47:15 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=288616
12/07/03 04:47:15 INFO mapred.JobClient:   File Input Format Counters
12/07/03 04:47:15 INFO mapred.JobClient:     Bytes Read=661630
12/07/03 04:47:15 INFO mapred.JobClient:   Map-Reduce Framework
12/07/03 04:47:15 INFO mapred.JobClient:     Map output materialized bytes=267085
12/07/03 04:47:15 INFO mapred.JobClient:     Combine output records=18040
12/07/03 04:47:15 INFO mapred.JobClient:     Map input records=12761
12/07/03 04:47:15 INFO mapred.JobClient:     Physical memory (bytes) snapshot=183209984
12/07/03 04:47:15 INFO mapred.JobClient:     Spilled Records=18040
12/07/03 04:47:15 INFO mapred.JobClient:     Map output bytes=1086716
12/07/03 04:47:15 INFO mapred.JobClient:     CPU time spent (ms)=1940
12/07/03 04:47:15 INFO mapred.JobClient:     Total committed heap usage  (bytes)=162856960
12/07/03 04:47:15 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=393482240
12/07/03 04:47:15 INFO mapred.JobClient:     Combine input records=109844
12/07/03 04:47:15 INFO mapred.JobClient:     Map output records=109844
12/07/03 04:47:15 INFO mapred.JobClient:     SPLIT_RAW_BYTES=114

有什么线索吗？

4

2 回答 2

1

为了像我一样在互联网上搜索并访问此页面的人的利益，您可能会在这里遇到 2 个问题

  1. DNS 解析 - 确保在安装 hadoop 时为每个主机使用完全限定的域名

  2. 防火墙 - 防火墙可能会阻止端口 50060,50030 以及基于您的 hadoop 分布的更多端口（cloudera 为 7182,7180）

于 2013-12-18T06:05:17.177 回答
0

我之前遇到过这个错误，它是由 DNS 问题引起的。您是否在基于 Linux 的发行版中运行？如果是这样，请确保您的所有 /etc/hosts 都是同步的，在我的情况下，我为每个节点“slave1 192.168.1.23”等使用了一个别名......但这与盒子名称不匹配，所以我不得不更改它，或者您可以更改您的盒子名称以匹配hadoop conf中“奴隶”对应的名称。

于 2013-01-28T03:16:16.437 回答