1

我正在使用 python 的 Requests 库从 BestBuy Products API 下载一些数据，我想将它们存储到 pandas 数据框中。

像这样的东西：

results = requests.get(url1, 
                  params={'paramStuff'}, 
                  headers={'User-Agent': ua})
products = json.loads(results.text)

获得了许多带有服务信息的各种字段，因此我只针对我想要的 JSON 中的特定字段：

products['products']

我有：

[{'details':[{'name': 'Name of Feature', 'value':'Value Of Feature'},
             {'name': 'Name of Other Feature', 'value':'Value Of Other 
              Feature'}, ...],
   'ProductId': 'Id Of Product 1',
   'Some Other Field': 'Some Other Field Value'},
 {same structure as above for other product}, {etc}]

因此，如您所见，它类似于字典列表，而字典列表本身又包含字典列表。突出显示 - details dict 可以有各种名称组合列表：值（名称也因产品而异）。

关于如何处理这种结构以进入具有这种格式的数据框的任何想法：

+-----------+-------------------+-------------------+-------------------+------------------+
| ProductID | Name of Feature 1 | Name of Feature 2 | Name Of Feature 3 | Some Other Field |
+-----------+-------------------+-------------------+-------------------+------------------+
| Product 1 | Value             | NULL              | Value             | Value            |
| Product 2 | NULL              | Value             | Value             | Value            |
+-----------+-------------------+-------------------+-------------------+------------------+

到目前为止，我只设法做到了这样的事情：

+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+
| ProductID |                                                              Details                                                              | Some Other Field |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+
| Product 1 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 1          |
| Product 2 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 2          |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+
4

1 回答 1

1

好的，我最终开发了一种手动解析嵌套字段的方法。不知道有没有什么简单的方法。仅供参考，这用于解析 BestBuy Products API 的响应，以防有人发现它有用。

#first build the pandas DF shown in question
df = pd.io.json.json_normalize(products)

#fields which are not nested and not require parsing
fields = ['sku', 'name', 'regularPrice', 'manufacturer']
#nested field is called 'details', as mentioned can have a lot of different subfields
featureFields = []


#first build a list which will have all potential features from the nested field
for i in range(0,len(df)):
    row = df.iloc[i]
    for detail in row['details']:
        featureFields.append(detail['name'].split('>', 1)[-1])

#make a list unique
featureFields = set(featureFields)      
fields = set(fields)

#now we go over each record in dataframe and parse nested field to a dict
records = []

for i in range(0,len(df)):
    row = df.iloc[i]
    record = dict.fromkeys(fields)
    record['name'] = row['name']
    record['regularPrice'] = row['regularPrice']
    record['manufacturer'] = row['manufacturer']
    record['sku'] = row['sku']
    for detail in row['details']:
        record[detail['name'].split('>', 1)[-1]] = detail['value'].split('>', 1)[-1]
    records.append(record)

#finally we have not nested list of dictionaries with records
dfFinal = pd.DataFrame.from_dict(records)
于 2017-06-22T02:18:10.730 回答