6

I have a DataFrame that includes a column where every cell is made up of a list of dicts, and each list of dicts is of varying length (including 0).

An example:

df = pd.DataFrame({'ID' : [13423,294847,322844,429847], 'RANKS': [[{u'name': u'A', u'price': u'$1.00', u'rank': u'1'},
{u'name': u'B', u'price': u'$4.00', u'rank': u'2'},
{u'name': u'C', u'price': u'$3.99', u'rank': u'3'},
{u'name': u'D', u'price': u'$2.00', u'rank': u'4'},
{u'name': u'E', u'price': u'$2.50', u'rank': u'5'}],

[{u'name': u'AA', u'price': u'$1.99', u'rank': u'1'},
{u'name': u'BB', u'price': u'$6.99', u'rank': u'2'}],

[{u'name': u'Z', u'price': u'$0.99', u'rank': u'1'},
{u'name': u'Y', u'price': u'$10.00', u'rank': u'2'},
{u'name': u'X', u'price': u'$1.99', u'rank': u'3'}],[]], 'count' : [5,2,3,0]})

Note that 'count' is the number of dicts in 'RANKS.' The goal I had in mind was to creating a series of additional dataframes/tables (one for each 'rank') and link these to the main table in a HDFStore. Something like:

Rank_2
ID       Price   Name
13423    $4.00    B  
294847   $6.99    BB 
322844   $10.99   Y 
429847   NaN      NaN   


Rank_3
ID       Price   Name
13423    $3.99    C  
294847   NaN      NaN 
322844   $1.99    X 
429847   NaN      NaN

This way I could easy query on ID and rank if needed, but the main table wouldn't get cluttered with the unwinding of this hierarchical data.

The problem, however, is I cannot figure out how to create the DataFrames from this column. I've tried a number of things, the first (to be nested in a for-loop if it worked, but of course it did not):

Rank_1 = pd.DataFrame(df.loc[df['count'] > 0]['RANKS'].map(lambda x: pd.DataFrame(x[0])))

And, the second, since price is the most important piece to me:

for i in range(0,5):
    df['rank_%s' % str(i+1)] = df[df['count'] > i]['RANKS'].map(lambda x: x[i]['price'].strip('$'))

Then convert to float. This works but is a pretty big compromise. Is there an efficient way (that won't get hung up on NaNs) to accomplish my goal of separate DataFrames for each rank?

4

3 回答 3

6

我的直觉反应是，您可能不应该将 DataFrame 分解为许多较小的 DataFrame。处理大量小型 DataFrame 需要一个 Python 循环，这通常是通往缓慢之路的一步。相反，我认为使用一个 DataFrame 可能会更好地为您服务，它会使 dicts 列表变平，以便每个内部 dict 在 DataFrame 中都有自己的行。内部字典的键将成为新列。我怀疑这种单一的平面 DataFrame 格式将能够执行多个 DataFrame 替代方案可以做的任何事情，但速度更快，并且它会使保存到 HDFStore 变得简单。

假设您有一个 DataFrame，列中有一个 dicts 列表RANKS：

import numpy as np
import pandas as pd

df = pd.DataFrame({'ID' : [13423,294847,322844,429847], 'RANKS': [[{u'name': u'A', u'price': u'$1.00', u'rank': u'1'},
{u'name': u'B', u'price': u'$4.00', u'rank': u'2'},
{u'name': u'C', u'price': u'$3.99', u'rank': u'3'},
{u'name': u'D', u'price': u'$2.00', u'rank': u'4'},
{u'name': u'E', u'price': u'$2.50', u'rank': u'5'}],

[{u'name': u'AA', u'price': u'$1.99', u'rank': u'1'},
{u'name': u'BB', u'price': u'$6.99', u'rank': u'2'}],

[{u'name': u'Z', u'price': u'$0.99', u'rank': u'1'},
{u'name': u'Y', u'price': u'$10.00', u'rank': u'2'},
{u'name': u'X', u'price': u'$1.99', u'rank': u'3'}],[]], 'count' : [5,2,3,0]})

然后您可以构建一个平面 DataFrame，每行一个字典，如下所示：

result = []
for idx, row in df.iterrows():
    for dct in row['RANKS']:
        dct['ID'] = row['ID']
        dct['count'] = row['count']
        result.append(dct)
del df
result = pd.DataFrame(result)
result['rank'] = result['rank'].astype(np.int32)
result['price'] = result['price'].str.replace('$', '')
result['price'] = result['price'].astype('float')
print(result)

产生

       ID  count name  price  rank
0   13423      5    A   1.00     1
1   13423      5    B   4.00     2
2   13423      5    C   3.99     3
3   13423      5    D   2.00     4
4   13423      5    E   2.50     5
5  294847      2   AA   1.99     1
6  294847      2   BB   6.99     2
7  322844      3    Z   0.99     1
8  322844      3    Y  10.00     2
9  322844      3    X   1.99     3

请注意，result直接从原始数据源构建（因此df完全避免）将是一种更清洁、内存需求更少的解决方案。

于 2015-02-22T20:20:16.497 回答
3

在 Pandas 版本0.25.0中，有df.explode列表爆炸的方法和一些字典爆炸的小代码。

如果您的数据框是：

import numpy as np
import pandas as pd

df = pd.DataFrame({'ID' : [13423,294847,322844,429847], 'RANKS': [[{u'name': u'A', u'price': u'$1.00', u'rank': u'1'},
{u'name': u'B', u'price': u'$4.00', u'rank': u'2'},
{u'name': u'C', u'price': u'$3.99', u'rank': u'3'},
{u'name': u'D', u'price': u'$2.00', u'rank': u'4'},
{u'name': u'E', u'price': u'$2.50', u'rank': u'5'}],

[{u'name': u'AA', u'price': u'$1.99', u'rank': u'1'},
{u'name': u'BB', u'price': u'$6.99', u'rank': u'2'}],

[{u'name': u'Z', u'price': u'$0.99', u'rank': u'1'},
{u'name': u'Y', u'price': u'$10.00', u'rank': u'2'},
{u'name': u'X', u'price': u'$1.99', u'rank': u'3'}],[]], 'count' : [5,2,3,0]})

然后要分解列表，您可以执行以下操作：

df = df.explode('RANKS')

这给了你

    ID  RANKS   count
0   13423   {'name': 'A', 'price': '$1.00', 'rank': '1'}    5
0   13423   {'name': 'B', 'price': '$4.00', 'rank': '2'}    5
0   13423   {'name': 'C', 'price': '$3.99', 'rank': '3'}    5
0   13423   {'name': 'D', 'price': '$2.00', 'rank': '4'}    5
0   13423   {'name': 'E', 'price': '$2.50', 'rank': '5'}    5
1   294847  {'name': 'AA', 'price': '$1.99', 'rank': '1'}   2
1   294847  {'name': 'BB', 'price': '$6.99', 'rank': '2'}   2
2   322844  {'name': 'Z', 'price': '$0.99', 'rank': '1'}    3
2   322844  {'name': 'Y', 'price': '$10.00', 'rank': '2'}   3
2   322844  {'name': 'X', 'price': '$1.99', 'rank': '3'}    3
3   429847  NaN 0

要分解这些 dicts 并将它们扩展为列，您可以执行以下操作：

df.reset_index(drop=True, inplace=True)

# Replace NaN by empty dict
def replace_nans_with_dict(series):
    for idx in series[series.isnull()].index:
        series.at[idx] = {}
    return series



# Explodes list and dicts
def df_explosion(df, col_name:str):

    if df[col_name].isna().any():
        df[col_name] = replace_nans_with_dict(df[col_name])

    df.reset_index(drop=True, inplace=True)

    df1 = pd.DataFrame(df.loc[:,col_name].values.tolist())

    df = pd.concat([df,df1], axis=1)

    df.drop([col_name], axis=1, inplace=True)

    return df

跑

df = df_explosion(df, 'RANKS')

你将拥有：

ID  count   name    price   rank
0   13423   5   A   $1.00   1
1   13423   5   B   $4.00   2
2   13423   5   C   $3.99   3
3   13423   5   D   $2.00   4
4   13423   5   E   $2.50   5
5   294847  2   AA  $1.99   1
6   294847  2   BB  $6.99   2
7   322844  3   Z   $0.99   1
8   322844  3   Y   $10.00  2
9   322844  3   X   $1.99   3
10  429847  0   NaN NaN NaN
于 2019-09-06T10:04:43.877 回答
1

我刚刚遇到了类似的情况，这就是我最终解决它的方法：

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({
   ...:     'ID' : [13423,294847,322844,429847],
   ...:     'RANKS': [[{u'name': u'A', u'price': u'$1.00', u'rank': u'1'},
   ...:                {u'name': u'B', u'price': u'$4.00', u'rank': u'2'},
   ...:                {u'name': u'C', u'price': u'$3.99', u'rank': u'3'},
   ...:                {u'name': u'D', u'price': u'$2.00', u'rank': u'4'},
   ...:                {u'name': u'E', u'price': u'$2.50', u'rank': u'5'}],
   ...: 
   ...:               [{u'name': u'AA', u'price': u'$1.99', u'rank': u'1'},
   ...:                {u'name': u'BB', u'price': u'$6.99', u'rank': u'2'}],
   ...: 
   ...:               [{u'name': u'Z', u'price': u'$0.99', u'rank': u'1'},
   ...:                {u'name': u'Y', u'price': u'$10.00', u'rank': u'2'},
   ...:                {u'name': u'X', u'price': u'$1.99', u'rank': u'3'}],[]]})

In [3]: import itertools

In [4]: temp_df = pd.DataFrame(
   ...:     list(itertools.chain(*[zip([key]*len(val), val)
   ...:                            for key, val in df.RANKS.iteritems()])),
   ...:     columns=['idx', 'explode'])                  

In [5]: exploded = pd.merge(
   ...:     df.drop('RANKS', axis=1),
   ...:     temp_df.explode.apply(pd.Series).join(temp_df.idx),
   ...:     left_index=True,
   ...:     right_on='idx',
   ...:     how='left').drop('idx', axis=1)

分解后的数据框如下所示：

In [6]: exploded
Out[6]: 
       ID name   price rank
0   13423    A   $1.00    1
1   13423    B   $4.00    2
2   13423    C   $3.99    3
3   13423    D   $2.00    4
4   13423    E   $2.50    5
5  294847   AA   $1.99    1
6  294847   BB   $6.99    2
7  322844    Z   $0.99    1
8  322844    Y  $10.00    2
9  322844    X   $1.99    3
9  429847  NaN     NaN  NaN
于 2016-12-12T01:10:38.507 回答