java - 如何对hadoop中的所有键/值对进行一般减少

Question

我是 hadoop 新手，我尝试在 Java 中执行一些 map/reduce 任务。我想知道我们如何对所有键/值对执行 reduce 操作。

例如，假设我们在一个月的每一天都有当天的最高温度。我们以天为键，以温度为值，我希望获得整个月最高温度的键/值。

我希望我的问题很清楚！

谢谢您的帮助。

Answer 1

Yes, it's possible. Just configure your job to use a single reducer via job.setNumReduceTasks(1). This single reducer will iterate over all key/value pairs. In the reduce() method you just search for the maximum and in the cleanup() method you output the final result. Example with (k, v) = (year, temperature):

public class MaxTemperatureReducer extends Reducer<IntWritable, DoubleWritable, IntWritable, DoubleWritable> {
    private static int year = 0;
    private static double maxTemp = 0.0;

    @Override
    public void reduce(IntWritable key, Iterable<DoubleWritable> values, Context context) throws IOException, InterruptedException {
        for (DoubleWritable value : values) {
            if (value.get() > maxTemp) {
                year = key.get();
                maxTemp = value.get();
            }
        }
    }

    @Override
    public void cleanup(Context context) throws IOException, InterruptedException {
        context.write(new IntWritable(year), new DoubleWritable(maxTemp));
    }
}

Answer 2

简单的方法是简单地使用任意键（“月”）并在值中同时包含温度和日期 - 然后在您的 reduce 方法中，找到温度的最高值并返回日期和温度。