hadoop - Reducers 如何在 Hadoop 中收集地图输出

Question

减速器是拉地图输出还是地图将输出推到减速器？如果 Reducers 拉动，那么他们如何知道要联系哪些 Maps（从属机器）。JobTracker 会告诉 Reducers 联系哪台机器吗？

Answer 1

This happens completely under the hood. Essentially the MapReduce Framework initiates a shuffle and sort which happens over the network and pass it to the reducers. Your Reducer or Mapper dont do any push or pull, its the Map reduce framework does it and also if you have your custom partitioner, comparator or group comparator it does the shuffling and sorting of data according to those before pushing it to the reducers.

I hope it helps.