1

我在 IPython 中运行的 shell 脚本返回以下对象：

results = ['{"url": "https://url.com", "date": "2020-10-02T21:25:20+00:00", "content": "mycontent\nmorecontent\nmorecontent", "renderedContent": "myrenderedcontent", "id": 123, "username": "somename", "user": {"username": "somename", "displayname": "some name", "id": 123, "description": "my description", "rawDescription": "my description", "descriptionUrls": [], "verified": false, "created": "2020-02-00T02:00:00+00:00", "followersCount": 1, "friendsCount": 1, "statusesCount": 1, "favouritesCount": 1, "listedCount": 1, "mediaCount": 1, "location": "", "protected": false, "linkUrl": null, "linkTcourl": null, "profileImageUrl": "https://myprofile.com/mypic.jpg", "profileBannerUrl": "https://myprofile.com/mypic.jpg"}, "outlinks": [], "outlinks2": "", "outlinks3": [], "outlinks4": "", "replyCount": 0, "retweetCount": 0, "likeCount": 0, "quoteCount": 0, "conversationId": 123, "lang": "en", "source": "<a href=\\"mysource.com" rel=\\"something\\">Sometext</a>", "media": [{"previewUrl": "smallpic.jpg", "fullUrl": "largepic.jpg", "type": "photo"}], "forwarded": null, "quoted": null, "mentionedUsers": [{"username": "name1", "displayname": "name 1", "id": 345, "description": null, "rawDescription": null, "descriptionUrls": null, "verified": null, "created": null, "followersCount": null, "friendsCount": null, "statusesCount": null, "favouritesCount": null, "listedCount": null, "mediaCount": null, "location": null, "protected": null, "linkUrl": null, "link2url": null, "profileImageUrl": null, "profileBannerUrl": null}]}', ...]

而...表示与前一个类似的更多条目。根据 type()，这是一个 slist。根据上述shell脚本的文档，这是一个jsonlines文件。

最终，我想将其转换为 csv 对象，其中键是列，值是值，其中每个条目（如上所示）是一行。所以像：

url              date                       content   ...
https://url.com  2020-10-02T21:25:20+00:00  mycontent ...

我已经尝试过这里提出的解决方案，但我收到了一个带有键值对的数据框，如下所示：

import pandas as pd
df = pd.DataFrame(data=results)
df = df[0].str.split(',',expand=True)
df = df.rename(columns=df.iloc[0])
4

1 回答 1

1

尽管您的示例数据包含几个问题，但如果您修复这些问题，则可以：

import json
import pandas as pd

fragment = '{"url": "https://url.com", "date": "2020-10-02T21:25:20+00:00", "content": "mycontent\\\\nmorecontent\\\\nmorecontent", "renderedContent": "myrenderedcontent", "id": 123, "username": "somename", "user": {"username": "somename", "displayname": "some name", "id": 123, "description": "my description", "rawDescription": "my description", "descriptionUrls": [], "verified": false, "created": "2020-02-00T02:00:00+00:00", "followersCount": 1, "friendsCount": 1, "statusesCount": 1, "favouritesCount": 1, "listedCount": 1, "mediaCount": 1, "location": "", "protected": false, "linkUrl": null, "linkTcourl": null, "profileImageUrl": "https://myprofile.com/mypic.jpg", "profileBannerUrl": "https://myprofile.com/mypic.jpg"}, "outlinks": [], "outlinks2": "", "outlinks3": [], "outlinks4": "", "replyCount": 0, "retweetCount": 0, "likeCount": 0, "quoteCount": 0, "conversationId": 123, "lang": "en", "source": "<a href=\\"mysource.com\\" rel=\\"something\\">Sometext</a>", "media": [{"previewUrl": "smallpic.jpg", "fullUrl": "largepic.jpg", "type": "photo"}], "forwarded": null, "quoted": null, "mentionedUsers": [{"username": "name1", "displayname": "name 1", "id": 345, "description": null, "rawDescription": null, "descriptionUrls": null, "verified": null, "created": null, "followersCount": null, "friendsCount": null, "statusesCount": null, "favouritesCount": null, "listedCount": null, "mediaCount": null, "location": null, "protected": null, "linkUrl": null, "link2url": null, "profileImageUrl": null, "profileBannerUrl": null}]}'

data = json.loads(fragment)
df = pd.DataFrame([data])
df.to_csv('test_out.csv')

注意：本例中的示例数据已修复，更改：

  • "在“源”中正确转义
  • \n被转义为\\\\n, 可能也是\\n如此，但我不认为你想要你的 csv 中的换行符

如果 results 是这些列表：

import json
import pandas as pd

results = get_results_somewhere()

df = pd.DataFrame([json.loads(r) for r in results])
df.to_csv('test_out.csv')

如果您输入的错误仅限于上述情况，您可以像这样修复它们：

def fix_input(s):
    return regex.sub('(?<=<[^>]*?)(")', r'\\"', regex.sub(r'(?<=<[^>]*?)(\\)', '', regex.sub('\n', '\\\\\\\\n', s)))

这将取消先前转义的\\"内部<>，然后将所有"内部<>替换\\"为它还“修复”换行符。如果您无法理解为什么正则表达式会以它们的方式工作，那可能是一个单独的问题。

整个东西：

import json
import regex
import pandas as pd


def fix_input(s):
    return regex.sub('(?<=<[^>]*?)(")', r'\\"', regex.sub(r'(?<=<[^>]*?)(\\)', '', regex.sub('\n', '\\\\\\\\n', s)))


results = get_results_somewhere()
fixed_results = fix_input(results)

df = pd.DataFrame([json.loads(r) for r in fixed_results])
df.to_csv('test_out.csv')

注意：这使用了第三方regex，而不是re因为它使用可变长度的lookbehind。

于 2020-10-02T22:32:25.843 回答