hadoop - Hadoop CombineFileInputFormatInput

Question

在输入格式中使用 CombineFileInputFormatInput 时出现错误 InstantiationException。

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://master:54310");

Job job = new Job(conf, "WordCount");
job.setJarByClass(WordCount.class);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setInputFormatClass(CombineFileInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path("/user/hduser/CombineFileInputFormatInput"));
FileOutputFormat.setOutputPath(job, new Path("/user/hduser/CombineFileInputFormatOutputNew2"));

System.exit(job.waitForCompletion(true) ? 0 : 1);

Answer 1

CombineFileInputFormat是一个抽象类，你必须扩展它并实现 createRecordReader 方法：

/**
 * This is not implemented yet. 
 */
public abstract RecordReader<K, V> createRecordReader(InputSplit split,
  TaskAttemptContext context) throws IOException;

然后用job.setInputFormatClass(..)扩展类的名称替换调用。