python - s3选择到熊猫数据框

Question

我正在使用 S3 Select 读取 csv 文件并输出到 JSON。现在我希望将 S3 Select 的 JSON 输出转换为 pandas 数据框。是否可以将 S3 Select JSON 输出转换为 pandas 数据框？

Answer 1

不确定您是否仍在寻找答案，但这对我有用：

s3 = boto3.client('s3')
bucket = bucket_name

file_name = file_key

sql_stmt = """SELECT S.* FROM s3object S LIMIT 10"""


req = s3.select_object_content(
    Bucket=bucket_name,
    Key=file_key,
    ExpressionType='SQL',
    Expression=sql_stmt,
    InputSerialization = {'CSV': {'FileHeaderInfo': 'USE'}},
    OutputSerialization = {'CSV': {}},
)

records = []
for event in req['Payload']:
    if 'Records' in event:
        records.append(event['Records']['Payload'])
    elif 'Stats' in event:
        stats = event['Stats']['Details']


file_str = ''.join(r.decode('utf-8') for r in records)

select_df = pd.read_csv(StringIO(file_str))
print(len(select_df))
df = pd.DataFrame(select_df)
print(df)