1

我一直在玩 MRUnit，并尝试按照wordcount和unit testing教程运行它以获得 hadoop wordcount 示例

虽然不是粉丝，但我一直在使用 Eclipse 运行代码，但我不断收到 setMapper 函数的错误

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;


import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;

import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;

import org.junit.Before;
import org.junit.Test;

public class TestWordCount {
  MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
  MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
  ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;

  @Before
  public void setUp() throws IOException
  {
      WordCountMapper mapper = new WordCountMapper();
      mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
      mapDriver.setMapper(mapper);  //<--Issue here

      WordCountReducer reducer = new WordCountReducer();
      reduceDriver = new ReduceDriver<Text, IntWritable, Text, IntWritable>();
      reduceDriver.setReducer(reducer);

      mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text, IntWritable,     Text, IntWritable>();
      mapReduceDriver.setMapper(mapper); //<--Issue here
      mapReduceDriver.setReducer(reducer);
  }

错误信息：

java.lang.Error: Unresolved compilation problems: 
    The method setMapper(Mapper<LongWritable,Text,Text,IntWritable>) in the type MapDriver<LongWritable,Text,Text,IntWritable> is not applicable for the arguments (WordCountMapper)
    The method setMapper(Mapper<LongWritable,Text,Text,IntWritable>) in the type MapReduceDriver<LongWritable,Text,Text,IntWritable,Text,IntWritable> is not applicable for the arguments (WordCountMapper)

查找此问题，我认为这可能是 API 冲突，但我不确定在哪里寻找它。其他人以前有这个问题吗？

编辑我正在使用用户定义的库，其中包含 hadoop2 jar 和最新的 Junit(4.10) jar。

编辑 2这是 WordCountMapper 的代码

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable> 
{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();


    public void map(Object key, Text value, Context context)throws IOException, InterruptedException 
    {
        StringTokenizer itr = new StringTokenizer(value.toString());
        while (itr.hasMoreTokens()) 
        {
            word.set(itr.nextToken());
            context.write(word, one);
        }
    }
}

最终编辑/它的作品

原来我需要设置

WordCountMapper mapper = new WordCountMapper();

至

Mapper mapper = new WordCountMapper();

因为泛型存在问题。还需要将 mockito 库导入我的用户定义库。

4

2 回答 2

2

这是你的问题

public class WordCountMapper extends Mapper<Object, Text, Text, IntWritable>
....
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;

您的WordCountMapper输入类型 ( ) 与输入类型( Object) 不兼容。将您的定义更改为MapDriverLongWritableMapper

class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>

您可能希望将您的map方法参数从更改Object key为LongWritable key也。

于 2014-06-09T14:02:24.757 回答
2

确保您已导入正确的类，我遇到了与上面不同的相同错误，我的程序在 Reducer 和 reduce_test 两个类中都有正确的参数，但是由于导入错误的类，我遇到了上面报告的相同错误消息

错误导入的类——

导入 org.apache.hadoop.mrunit.ReduceDriver；

正确的类---

导入 org.apache.hadoop.mrunit.mapreduce.ReduceDriver；

mapper_test 的相同解决方案，如果您确定 Mapper__class 和 Mapper_test 中的参数相同

于 2015-12-06T10:20:42.413 回答