2 
   I am new to BigData I have written a MR program and trying to write test cases for it using MRUnit from https://dzone.com/articles/testing-mapreduce-mrunit

但是我的 MR 程序有 2 个映射器和 1 个减速器，所以我无法使用创建驱动程序对象

newMapReduceDriver()          
newMapReduceDriver(mapper,reducer)           
newMapReduceDriver(mapper,reducer,combiner)

或者

newMultipleInputMapReduceDriver()
newMultipleInputMapReduceDriver(combiner,reducer)
newMultipleInputMapReduceDriver(reducer)

请建议我做错事的任何其他方式。在此先感谢

这是代码

public class UC_RJoinerTool extends Configured implements Tool{

    public int run(String[] args) throws Exception {
        if(args == null || args.length < 4 ){
            System.err.println("Usage: <User file Input Path> <Comments file Input Path> <Output Path> <Inner/Right Outer/Full join>");
            ToolRunner.printGenericCommandUsage(System.err);
            return -1;
        }  else {

            Job job = Job.getInstance(getConf(), "Mapping Users with Comments");
            job.setJarByClass(UC_RJoinerTool.class);
            Path userInputPath = new Path(args[0]);
            Path commentsInputPath = new Path(args[1]);
            Path outPutPath = new Path(args[2]);
            String joinTypeInput = args[3];

            MultipleInputs.addInputPath(job, userInputPath, TextInputFormat.class,UserDotXmlMapper.class);
            MultipleInputs.addInputPath(job, commentsInputPath, TextInputFormat.class,CommentsDotXmlMapper.class);
            FileOutputFormat.setOutputPath(job,new Path(args[1]));
            //When you are using TextInputFormat explicitly say the map key and value types
            job.setMapOutputKeyClass(Text.class);
            job.setMapOutputValueClass(Text.class);

            job.getConfiguration().set("joinType",joinTypeInput);
            job.setReducerClass(UserCommentsReducer.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(Text.class);
            FileOutputFormat.setOutputPath(job, outPutPath);

            return job.waitForCompletion(true)?0:1;
        }
    }

    public static void main(String[] args)throws Exception {
        int exitCode = ToolRunner.run(new UC_RJoinerTool(),args);
        System.exit(exitCode);
    }
}

我的单元测试用例代码

public class UCJoinTest {
    private MapDriver<LongWritable,Text,Text,Text> mapUsersDriver,mapCommentsDriver;    
    private ReduceDriver<Text, Text, Text,Text> reduceUCDriver;
    //private MapReduceDriver<LongWritable,Text,Text, Text, Text,Text> mapReduceUCDriver;
    private MultipleInputsMapReduceDriver<LongWritable,Text,Text,Text> mapReduceUCDriver;

private MapDriver<LongWritable, Text, LongWritable,Text> mapSortDriver;
private ReduceDriver<LongWritable,Text,Text,Text> reduceSortDriver;
private MapReduceDriver<LongWritable,Text,LongWritable,Text,Text,Text> mapReduceSortDriver;

@Before
public void setUp() throws Exception {
    final UserDotXmlMapper usersMapper = new UserDotXmlMapper();
    final CommentsDotXmlMapper CommentsMapper = new CommentsDotXmlMapper();
    final UserCommentsReducer ucReducer = new UserCommentsReducer();

    final ReputationSorterMapper sortMapper = new ReputationSorterMapper();
    final ReputationSorterReducer sortReducer = new ReputationSorterReducer();

    mapUsersDriver = MapDriver.newMapDriver(usersMapper);
    mapCommentsDriver = MapDriver.newMapDriver(CommentsMapper);     
    reduceUCDriver = ReduceDriver.newReduceDriver(ucReducer);

    mapReduceUCDriver = MapReduceDriver.newMapReduceDriver(usersMapper,CommentsMapper,ucReducer);
    mapReduceUCDriver = MultipleInputsMapReduceDriver.newMultipleInputMapReduceDriver(usersMapper,CommentsMapper,ucReducer);         

    mapSortDriver = MapDriver.newMapDriver(sortMapper); 
    reduceSortDriver = ReduceDriver.newReduceDriver(sortReducer);
    mapReduceSortDriver = MapReduceDriver.newMapReduceDriver(sortMapper,sortReducer);
}


public class CommentsDotXmlMapper extends Mapper<LongWritable,Text,Text,Text>{
}
public class UserDotXmlMapper extends Mapper<LongWritable,Text,Text,Text>{
}
public class UserCommentsReducer extends Reducer<Text, Text, Text,Text>{
}

出于某种原因，Stackowerflow 不允许我发布问题，所以我添加此评论请忽略此

4

0 回答 0