1

嗨，我发现 map reduce 链接有点问题。我必须形成这样的链

映射器->减速器->映射器

从我的第一个映射器到减速器，流程一直很好，并且这个减速器的输出数据不会正确地进入下一个映射器。这是我尝试过的一个简单的代码示例

这是我的第一个映射器

public void map(LongWritable key, Text value,
        OutputCollector<Text, IntWritable> outputCollector, Reporter reporter)
        throws IOException {

    String maxSalary = value.toString().split(",")[4]; 

    outputCollector.collect(new Text("max salary"),new IntWritable(Integer.parseInt(maxSalary)));

}

这是我的减速机

public void reduce(Text key, Iterator<IntWritable> values,
            OutputCollector<Text, IntWritable> outputCollector, Reporter reporter)
            throws IOException {
        int maxSalary = Integer.MIN_VALUE;

        while(values.hasNext()){

            maxSalary = Math.max(maxSalary, values.next().get());
        }
        outputCollector.collect(key, new IntWritable(maxSalary));

    }

这是我的下一个简单的映射器

public void map(Text key, IntWritable value,
            OutputCollector<Text, IntWritable> outputCollector, Reporter reporter)
            throws IOException {

        System.out.println(value.toString());
    }

这是我运行这项工作的主要课程

JobConf jobConf = new JobConf(jobrunner.class);
jobConf.setJobName("Chaining");

FileInputFormat.setInputPaths(jobConf, new Path("hdfs://localhost:9000/employee_data.txt"));
FileOutputFormat.setOutputPath(jobConf,new Path("hdfs://localhost:9000/chain9.txt"));

JobConf conf1 = new JobConf(false);

ChainMapper.addMapper(jobConf,chainmap.class,LongWritable.class,Text.class,Text.class,IntWritable.class,true,conf1);

JobConf conf2 = new JobConf(false);

ChainReducer.setReducer(jobConf, chainreduce.class,Text.class,IntWritable.class,Text.class,IntWritable.class,true,conf2);

JobConf conf3 = new JobConf(false);

ChainMapper.addMapper(jobConf, nextchainmap.class, Text.class,IntWritable.class,Text.class,IntWritable.class,true,conf3);


JobClient.runJob(jobConf);

我将在我的减速器中获得最高员工工资，这必须传递给下一个映射器，在那里它将找到具有最大工资值的员工记录，我如何在下一个映射器中实现这一点？有什么想法吗？

4

1 回答 1

2

要链接您的第二个映射器，您需要调用ChainReducer.addMapper(...)而不是ChainMapper.addMapper(...).

于 2013-08-18T18:16:04.160 回答