0

我正在生成一个 botocore 会话，该会话承担一个角色，该角色使我可以访问不同帐户中的 s3 存储桶。基于这个角色创建一个 s3fs 文件系统让我可以操作这些数据。这是相关的代码部分。role_session是具有适当凭据的 botocore 会话。

fs_session = role_session._session
fs = s3fs.S3FileSystem(session=fs_session)
bucket_name = "my_data_bucket"
print(fs.ls(bucket_name))

这适用于 s3fs = 0.4.2 的 SageMaker 笔记本实例。我们称这个实例为 A。

但是在具有最新版本 s3fs = 2021.07.0 的笔记本实例上，相同的代码会给出拒绝访问错误。让我们将此实例称为 B。这是完整的跟踪：

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/s3fs/core.py in _lsdir(self, path, refresh, max_items, delimiter, prefix)
    573                 dircache = []
--> 574                 async for i in it:
    575                     dircache.extend(i.get("CommonPrefixes", []))

/opt/conda/lib/python3.7/site-packages/aiobotocore/paginate.py in __anext__(self)
     31         while True:
---> 32             response = await self._make_request(current_kwargs)
     33             parsed = self._extract_parsed_response(response)

/opt/conda/lib/python3.7/site-packages/aiobotocore/client.py in _make_api_call(self, operation_name, api_params)
    154             error_class = self.exceptions.from_code(error_code)
--> 155             raise error_class(parsed_response, operation_name)
    156         else:

ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

The above exception was the direct cause of the following exception:

PermissionError                           Traceback (most recent call last)
<ipython-input-162-af0f8b8e4877> in <module>
      9 fs = s3fs.S3FileSystem(session=fs_session)
     10 bucket_name = "my_data_bucket"
---> 11 print(fs.ls(bucket_name))

/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py in wrapper(*args, **kwargs)
     86     def wrapper(*args, **kwargs):
     87         self = obj or args[0]
---> 88         return sync(self.loop, func, *args, **kwargs)
     89 
     90     return wrapper

/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py in sync(loop, func, timeout, *args, **kwargs)
     67         raise FSTimeoutError
     68     if isinstance(result[0], BaseException):
---> 69         raise result[0]
     70     return result[0]
     71 

/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py in _runner(event, coro, result, timeout)
     23         coro = asyncio.wait_for(coro, timeout=timeout)
     24     try:
---> 25         result[0] = await coro
     26     except Exception as ex:
     27         result[0] = ex

/opt/conda/lib/python3.7/site-packages/s3fs/core.py in _ls(self, path, detail, refresh)
    777             files = await self._lsbuckets(refresh)
    778         else:
--> 779             files = await self._lsdir(path, refresh)
    780             if not files and "/" in path:
    781                 files = await self._lsdir(self._parent(path), refresh=refresh)

/opt/conda/lib/python3.7/site-packages/s3fs/core.py in _lsdir(self, path, refresh, max_items, delimiter, prefix)
    595                     f["name"] = f["Key"]
    596             except ClientError as e:
--> 597                 raise translate_boto_error(e)
    598 
    599             if delimiter and files:

PermissionError: Access Denied

实例 A 和实例 B 之间还有其他版本差异，包括 python/botocore，但我不确定它们是否是一个问题。这是因为在实例 A 上升级 s3fs 会导致代码停止工作，而在实例 B 上降级 s3fs 会使代码工作。此外，在 botocore 之上创建 s3 客户端在任一版本中都可以正常工作。

除了 s3fs 降级之外，谁能指出如何解决这个问题？我无法真正避免 s3fs，因为它已用于其他各个部分。

4

0 回答 0