0

这类似于Amazon s3 with big files 上的 dask read_csv timeout，但这实际上并没有解决我的问题。

import s3fs
fs = s3fs.S3FileSystem()

fs.connect_timeout = 18000
fs.read_timeout = 18000 # five hours

fs.download('s3://bucket/big_file','local_path_to_file')

然后我得到的错误是

Traceback (most recent call last):
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/aiobotocore/response.py", line 50, in read
    chunk = await self.__wrapped__.read(amt if amt is not None else -1)
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/aiohttp/streams.py", line 380, in read
    await self._wait("read")
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/aiohttp/streams.py", line 306, in _wait
    await waiter
aiohttp.client_exceptions.ServerTimeoutError: Timeout on reading data from socket
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/fsspec/spec.py", line 1113, in download
    return self.get(rpath, lpath, recursive=recursive, **kwargs)
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/fsspec/asyn.py", line 281, in get
    return sync(self.loop, self._get, rpaths, lpaths)
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/fsspec/asyn.py", line 71, in sync
    raise exc.with_traceback(tb)
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/fsspec/asyn.py", line 55, in f
    result[0] = await future
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/fsspec/asyn.py", line 266, in _get
    return await asyncio.gather(
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/s3fs/core.py", line 701, in _get_file
    chunk = await body.read(2**16)
  File "/Users/christopherturnbull/PointTopic/PointTopic/lib/python3.9/site-packages/aiobotocore/response.py", line 52, in read
    raise AioReadTimeoutError(endpoint_url=self.__wrapped__.url,
aiobotocore.response.AioReadTimeoutError: Read timeout on endpoint URL: "https://ptpiskiss.s3.eu-west-1.amazonaws.com/REBUILD%20FOR%20TIME%20SERIES/v30a%20sept%202019.accdb"

这很奇怪，因为我认为我在类的工作副本上设置了适当的超时。这完全是因为我的互联网连接不好，但是我需要在我的 s3 端做些什么来帮助这里吗？

4

0 回答 0