hadoop - What can cause hadoop kill reducer task an retry

Question

my hadoop job has a very high ‘Killed Task Attempts’ number on its reducer tasks, I check the status of killed task:

Request received to kill task 'attempt_201308122006_41526_r_000030_1' by user
-------
Task has been KILLED_UNCLEAN by the user

and no stdout and stderr logs

what could cause this ? and how can I solve it?

score 3 · Accepted Answer

如果您启用了推测执行，那么您可能会看到许多将被“杀死”的 map/reduce 任务。这是由于 hadoop 在多个任务跟踪器上运行长时间运行的任务，并且第一个完成“胜利”而其他人被杀死。

一般来说，我只会担心在工作跟踪器中“失败”的任务尝试

尝试关闭推测执行：

mapred.map.tasks.speculative.execution = false mapred.reduce.tasks.speculative.execution = false

score 2 · Accepted Answer

如果不是推测执行，则可能是 Fair Scheduler 开始使用 minMaps 和 minReduces 为池声明任务跟踪器。

2 回答 2