1

My mapper emits 'uniq key' - 'very large value' pair.

My reducer doesn't know the key is unique. Thus, the reducer waits until all the mappers are completed.

I tried to use a combiner, but it is not an easy solution for me, because my reducer is very complicated.

My question is how can I perform the reducer after per map? without using a combiner.

4

3 回答 3

3

If your keys are uniq then the is no need to reduce them. Therefore just copy-paste reducer code to mapper and set reducer number to zero. BTW there are many map reduce jobs which do not require reduce step so it is not something strange.

于 2011-06-26T06:49:41.840 回答
2

If you know in advance your key is unique then you can move all the code from the reducer step into the map and to all the work there.

于 2011-06-26T08:08:47.827 回答
0

I don't understand your question. You can simply not specify a combiner in the Job configuration.

于 2011-06-25T17:53:20.947 回答