如果您提供
'folderA/folderB/folderC/mainfolder/*/*'
作为输入并希望过滤掉特定路径,您可能需要创建自定义 PathFilter
在FileInputFormat你有这个功能-
static void setInputPathFilter (JobConf conf, Class<? extends PathFilter> filter)
Info: Set a PathFilter to be applied to the input paths for the map-reduce job
例如
public static class CustomPathFilter implements PathFilter {
@Override
public boolean accept(Path path) {
//you can implement your logic for finding the valid range of paths here.
//The valid range of dates and days for directories and files can be input
//as arguments to the job.
//Return true if you find a match or else return false.
return false;
}
}
像这样注册 PathFilter -
FileInputFormat.setInputPathFilter(job, CustomPathFilter.class);