0

我希望能够在我的 MR 作业的映射阶段设置某种变量或标志，我可以在作业完成后检查这些变量或标志。我认为用一些代码来演示我想要什么的最好方法：ps 我正在使用 Hadoop 2.2.0

public class MRJob {

  public static class MapperTest 
       extends Mapper<Object, Text, Text, IntWritable>{


    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
        //Do some computation to get new value and key
        ...
        //Check if new value equal to some condition e.g if(value < 1) set global variable to true

        context.write(newKey, newValue);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();

    Job job = Job.getInstance(new Configuration(), "word_count");
   //set job configs

    job.waitForCompletion(true);

    //Here I want to be able to check if my global variable has been set to true by any one of the mappers

  }
}
4

1 回答 1

2

为此使用 a Counter。

public static enum UpdateCounter {
  UPDATED
}

 @Override
 public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

    if(value < 1) {
      context.getCounter(UpdateCounter.UPDATED).increment(1);
    }

    context.write(newKey, newValue);
}

工作后您可以检查：

Configuration conf = new Configuration();

Job job = Job.getInstance(new Configuration(), "word_count");
//set job configs

job.waitForCompletion(true);
long counter = job.getCounters().findCounter(UpdateCounter.UPDATED).getValue();   

if(counter > 0) 
  // some mapper has seen the condition
于 2014-07-16T17:26:13.170 回答