1

就像从 s3 中读取 spring 批处理中的单个文件一样，我们使用

@Bean
public FlatFileItemReader<Map<String, Object>> itemReader() {
    FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
    reader.setLineMapper(new JsonLineMapper());
    reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
    reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
    return reader;
}

但是，如果我想从某个特定文件夹/键中读取所有文件，那么 MultiResourceItemReader 是否有一些东西，如下所示（我们用于本地文件系统）

MultiResourceItemReader<UserData> reader = new MultiResourceItemReader<>();
reader.setResources(resources);
4

2 回答 2

2

像这样创建一个 MultiResourceItemReader，

@Autowired
private AmazonS3 s3;

@Autowired
private ResourceLoader resourceLoader;


public MultiResourceItemReader<String> fileItemReader() throws Exception {

    List<Resource> resourceList = new ArrayList<>();

    String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here

    //TODO: warn: this functn can only return max 1000 objects
    s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();

    for(S3ObjectSummary it:s3objects)
        resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));

    Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);

    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(flatFileItemReader());

    return reader;
}

这个读者需要一个委托和lineMapper，你可以像这样实现它，

private FlatFileItemReader<String> flatFileItemReader() throws Exception {

    FlatFileItemReader<String> reader = new FlatFileItemReader<>();
    JsonLineMapper lineMapper = new JsonLineMapper();
    reader.setLineMapper(lineMapper);
    reader.afterPropertiesSet();

    return reader;
}


public class JsonLineMapper implements LineMapper<String> {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public String mapLine(String s, int i) throws Exception {

        return s;
    }
}
于 2021-01-07T06:26:20.463 回答
2

不，由您来创建Resource数组并将其传递给MultiResourceItemReader.

于 2020-09-18T09:38:54.023 回答