java - map-reduce 代码的输出是什么？

Question

public static class MapClass extends MapReduceBase implements
        Mapper<LongWritable, Text, Text, IntWritable> {

    private Text word = new Text();

    public void map(LongWritable key, Text value, 
                    OutputCollector<Text, IntWritable> output, 
                    Reporter reporter) throws IOException {
      String line = value.toString();
      String num = Integer.parseInt(line);

       IntWritable one = new IntWritable(num);

        word.set(“key”);
        output.collect(word, one);

    }
}

public static class Reduce extends MapReduceBase implements
        Reducer<Text, IntWritable, Text, IntWritable> {

    public void reduce(Text key, Iterator<IntWritable> values,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum = 0;
        int count = 0;
        int avg = 0;
        while (values.hasNext()) {
            sum += values.next().get();
            count++;
        }
        avg = sum / count;
        output.collect(key, new IntWritable(count));
    }
}

具体请参见 output.collect()，我正在打印键和计数值。 对于任何输入文件，输出是 键 2 请帮助我....（即使输入为 100 个，输出如何始终为 2？？ )

Answer 1

如果你想测试你的代码，

这是一个使用 mockito 框架的 JUNIT 测试，假设你在示例类中有你的 reducer 和 mapper。

@RunWith(MockitoJUnitRunner.class)
 public class TestMapper {

 @Mock
 OutputCollector<Text, IntWritable> output;

 @Mock
 Reporter reporter;

 @Test
 public void testMap() throws Exception {
    ExampleClass.MapClass mapper = new ExampleClass.MapClass();
    mapper.map(new LongWritable(0), new Text("1"), output, reporter);
    mapper.map(new LongWritable(0), new Text("2"), output, reporter);
    mapper.map(new LongWritable(0), new Text("3"), output, reporter);
    verify(output, times(1)).collect(new Text("key"), new IntWritable(1));
    verify(output, times(1)).collect(new Text("key"), new IntWritable(2));
    verify(output, times(1)).collect(new Text("key"), new IntWritable(3));
 }

 @Test
 public void testReduce() throws Exception {
    ExampleClass.Reduce reducer = new ExampleClass.Reduce();
    List<IntWritable> list = Arrays.asList(new IntWritable(1), new IntWritable(2), new IntWritable(3));

    reducer.reduce(new Text("key"), list.iterator(), output, reporter);

    verify(output, times(1)).collect(new Text("key"), new IntWritable(3));
 }
}