0

When I checked the Hadoop GUI, I found that some of the reduce tasks have reached 66.66%, and they stay there for a long time. When I checked the counters, I found that the no. of input records is shown as zero.

After a long time, they get their input records, start processing them. Some show 0 input records in even for longer times and are killed by the Task Attempt failed to report status for 600 ms.

But some of the reducers show input records in their counters immediately and start processing them right away.

I do not know, why there is so much delay in the getting the input records for some reducers. This happens only with this program, and not the other programs.

In this mapreduce job, I have, in the configure method before the reduce method of the reduce, I read a lot of data from distributed cache. Is this the reason? I am not sure.

4

1 回答 1

1

是的，我相信从分布式缓存中读取是您延迟的原因。configure()但是如果你在 之前或之后保留它不会有什么不同reduce()，因为最终configure()方法必须首先被调用，如果你看到run()reducer 它看起来如下所示（新 API）：

public void run(Context context) throws IOException, InterruptedException {

    setup(context); // This is the counterpart of configure() from older API

    while (context.nextKey()) {
        reduce(context.getCurrentKey(), context.getValues(), context);
    }
    cleanup(context);
}

如您所见，在较旧的 API 中setup()调用reduce()了类似的方法，除非configure()完成实际的 reduce 任务，否则不会启动（这说明您看不到任何输入记录计数）。

现在至于百分比 : 66%，您会看到 reduce 阶段实际上具有以下子部分：

  1. 复制
  2. 种类
  3. 减少

因此，由于您的前两个步骤已完成，第三个步骤已开始但正在等待configure()完成（要读取分布式缓存），因此您的减少百分比为 66%。

于 2013-03-16T18:58:59.523 回答