hadoop - How to modify the Map-Reduce execution order?

Question

The original Map-Reduce execution chain is: InputSplits-->Mapper--> [Sorting/Shuffling, etc]-->Reducer-->...

Now I don't want the input splits to get to the Mappers first, but to go to some other new stage instead (we can call it Pre-Mapper for example, this class will be created by myself).

So the new order will be: InputSplits -> Pre-Mapper->Mapper ->...

I'm currently reading the source code. However, I still cannot find any clue (what classes I should touch).

Any suggestion is welcome. Thank you very much :)

Answer 1

Maybe you should take a look at chaining mappers: ChainMapper

Answer 2

you can implement it use two stages MapReduce:

stage one: Mapper -> Sorting/Shuffling -> Reducer[this reducer do nothing but write the data directly form Mapper];

stage two: Mapreduce;

this stage one here is what you do in your Pre Mapper;

Answer 3

You can consider overriding the MapRunner class