2

I have a map reduce job which uses a very large number of input paths. Consequently I am running into this exception while starting the job:

Exceeded max jobconf size: 6154861 limit: 5242880

I realized that the max jobconf size is governed by the system property:

mapred.user.jobconf.limit which defaults to 5 MB.

Is there a way to circumvent this issue other than increasing this limit ?

4

2 回答 2

1

或者，如果可能，您可以尝试使用DistributedCache。

您可以将它们放在JobConf一个文件中并将该文件添加到DistributedCache. 这取决于您的应用程序存储在JobConf.

于 2013-06-14T09:40:29.537 回答
1

我不这么认为，除非您准备好更改源本身。根据 JobTracker 的来源，如果用户未明确指定，这是允许的最大值。

/** the maximum allowed size of the jobconf **/
long MAX_JOBCONF_SIZE = 5*1024*1024L;

/** the config key for max user jobconf size **/
public static final String MAX_USER_JOBCONF_SIZE_KEY = "mapred.user.jobconf.limit";

如果您没有指定任何值，mapred.user.jobconf.limit则将5*1024*1024L使用：

MAX_JOBCONF_SIZE = conf.getLong(MAX_USER_JOBCONF_SIZE_KEY, MAX_JOBCONF_SIZE);
于 2013-06-12T23:23:14.620 回答