我已经实现了一个自定义的组合文件输入格式,以便为由一组文件组成的 Map 任务创建拆分。我创建了一个解决方案,通过记录阅读器传递拆分的每个文件,一切都很好。现在我试图将整个文件集传递给 map 函数。
这是我的记录阅读器代码:
public class MultiImagesRecordReader extends
RecordReader<Text[], BytesWritable[]> {
private long start = 0;
private long end = 0;
private int pos = 0;
private BytesWritable[] value;
private Text key[];
private CombineFileSplit split;
private Configuration conf;
private FileSystem fs;
private static boolean recordsRead;
public MultiImagesRecordReader(CombineFileSplit split,
TaskAttemptContext context, Integer index) throws IOException {
this.split = split;
this.conf = context.getConfiguration();
}
@Override
public void initialize(InputSplit genericSplit, TaskAttemptContext context)
throws IOException, InterruptedException {
start = split.getOffset(0);
end = start + split.getLength();
recordsRead = false;
this.pos = (int) start;
fs = FileSystem.get(conf);
value = new BytesWritable[split.getNumPaths()];
key = new Text[split.getNumPaths()];
}
@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if (recordsRead == true) {
System.out.println("Sono nel next true"+InetAddress.getLocalHost());
return false;
} else {
recordsRead = true;
System.out.println("Sono nel next false"+InetAddress.getLocalHost());
for (int i = 0; i < split.getNumPaths(); i++) {
int fileLength = (int) split.getLength(i);
Path path = split.getPath(i);
byte[] result = new byte[fileLength];
FSDataInputStream in = null;
String file_path = path.toString();
key[i] = new Text(file_path);
try {
in = fs.open(path);
IOUtils.readFully(in, result, 0, fileLength);
} finally {
IOUtils.closeStream(in);
}
value[i] = new BytesWritable(result);
}
return true;
}
}
使用此代码,map 函数会正确接收键和值的向量,但会重复。我的意思是,我希望 map 函数被调用一次,而不是被多次调用。我究竟做错了什么?