python - Pyarrow s3fs 按时间戳分区

Question

是否可以在将镶木地板文件写入时使用表中的时间戳字段通过“”对文件系统pyarrow进行分区？s3fsYYYY/MM/DD/HHs3

score 5 · Accepted Answer

我能够使用 pyarrow write_to_dataset 函数来实现，该函数允许您指定分区列来创建子目录。

例子：

import os
import s3fs
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from pyarrow.filesystem import S3FSWrapper

access_key = <access_key>
secret_key = <secret_key>
bucket_name = <bucket_name>

fs = s3fs.S3FileSystem(key=access_key, secret=secret_key)

bucket_uri = 's3://{0}/{1}'.format(bucket_name, "data")

data = {'date': ['2018-03-04T14:12:15.653Z', '2018-03-03T14:12:15.653Z', '2018-03-02T14:12:15.653Z', '2018-03-05T14:12:15.653Z'],
        'battles': [34, 25, 26, 57],
        'citys': ['london', 'newyork', 'boston', 'boston']}
df = pd.DataFrame(data, columns=['date', 'battles', 'citys'])
df['date'] = df['date'].map(lambda t: pd.to_datetime(t, format="%Y-%m-%dT%H:%M:%S.%fZ"))
df['year'], df['month'], df['day'] = df['date'].apply(lambda x: x.year), df['date'].apply(lambda x: x.month), df['date'].apply(lambda x: x.day)
df.groupby(by=['citys'])
table = pa.Table.from_pandas(df)
pq.write_to_dataset(table, bucket_uri, filesystem=fs, partition_cols=['year', 'month', 'day'], use_dictionary=True,  compression='snappy', use_deprecated_int96_timestamps=True)