尝试使用 fastparquet lib 在 S3 中附加 parquet 文件,出现以下错误:
文件“/Users/baluinfo/PycharmProjects/untitled/rough.py”,第 55 行,在 write(parqKey, ws1,write_index=False,append=True,compression='GZIP', open_with=myopen) 文件“/Users/baluinfo /PycharmProjects/Dash/venv/lib/python3.7/site-packages/fastparquet/writer.py”,第 881 行,在写入压缩、open_with、has_nulls、追加)文件“/Users/baluinfo/PycharmProjects/Dash/venv/ lib/python3.7/site-packages/fastparquet/writer.py”,第 735 行,在 write_simple 中,open_with(fn, mode) 为 f:文件“/Users/baluinfo/PycharmProjects/Dash/venv/lib/python3.7 /site-packages/fsspec/spec.py”,第 775 行,在打开的 **kwargs 文件中“/Users/baluinfo/PycharmProjects/Dash/venv/lib/python3.7/site-packages/s3fs/core.py”,第 378 行,在 _open autocommit=autocommit 中,requester_pays=requester_pays) 文件“/Users/baluinfo/PycharmProjects/Dash/venv/lib/python3.7/site-packages/s3fs/core.py”,第 1097 行,在init cache_type=cache_type) 文件 "/Users/baluinfo/PycharmProjects/Dash/venv/lib/python3.7/site-packages/fsspec/spec.py", 第 1062 行, init raise NotImplementedError("文件模式不支持") NotImplementedError:不支持文件模式在以下位置忽略异常:Traceback(最近一次调用最后一次):文件“/Users/baluinfo/PycharmProjects/Dash/venv/lib/python3.7/site-packages/fsspec/spec.py”,第 1343 行,在del self.close() 文件“/Users/baluinfo/PycharmProjects/Dash/venv/lib/python3.7/site-packages/fsspec/spec.py”中,第 1321 行,如果不是 self.forced,则关闭:AttributeError :“S3File”对象没有“强制”属性
aws_id = os.environ.get("AWS_ID")
aws_secret = os.environ.get("AWS_KEY")
bucket_name1 = 'eoddb/'
object_key = 'EOD.parquet'
s3 = s3fs.S3FileSystem(anon=False, key=aws_id, secret=aws_secret)
myopen = s3.open
parqKey = bucket_name1 + object_key
write(parqKey, ws1 ,write_index=False,append=True,compression='GZIP', open_with=myopen)