1

我经常遇到使用 AWS 的 IAM 用户缺少一项特定操作的问题。一个很好的例子是将数据上传到 S3。Hadoop 抛出以下异常：

2018-08-03 09:29:46,112 INFO [IPC Server handler 27 on 42415] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1526322305732_0008_m_000008_0 is : 0.0
2018-08-03 09:29:46,134 FATAL [IPC Server handler 26 on 42415] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1526322305732_0008_m_000008_0 - exited : com.cloudera.com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 008CAB66479B6842), S3 Extended Request ID: JtUlF07hBNr03NhytAQj6biGX8I/YKjtbUcz82PkjbLoDeoW3W8AVLvhAdXWk7V9Fc8G4oOy1d8=
    at com.cloudera.com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
    at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
    at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
    at com.cloudera.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
    at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
    at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1050)
    at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:961)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:78)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
    at org.apache.hadoop.tools.mapred.CopyMapper.setup(CopyMapper.java:114)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

此异常没有足够的信息来确定用户缺少哪个操作。确定必须为此类任务添加哪些操作的最佳方法是什么？

4

1 回答 1

2

我们经常使用 CloudTrail，您可以在其中查看尝试调用的内容以及响应是否成功以及原因是什么。

于 2018-08-09T11:32:32.503 回答