google-app-engine - mapreduce 的参数

Question

我正在使用 appengine 的 Java mapreduce 模块我收到以下信息消息

超出映射器配额。中止请求，直到配额被补充。如果您希望映射器作业更快完成，请考虑增加 mapreduce.mapper.inputprocessingrate（默认 1000）。

任务参数。
队列名称 = 默认
速率 = 1/s
bucketsize = 1

我有大约 2000 个 KIND 实体，我只是在 map() 调用中进行记录

需要提供哪些 mapreduce/task 参数来消除该信息消息。

-阿斯瓦特

Answer 1

I believe that this is a special quota in mapreduce implemented by the framework itself. It's designed to limit the speed that it can consume resources, so that mapreduce does not run through available app engine quota too quickly. It looks like it denotes the maximum overall rate of map() calls/second.

Try increasing the mapreduce.mapper.inputprocessingrate property in the configuration for your map job. Or, just to test, you can change the default, defined in mapreduce/AppEngineJobContext.java:

public static final int DEFAULT_MAP_INPUT_PROCESSING_RATE = 1000;