2

我有一个看起来“上三角”的数据框：

            31-May-11  30-Jun-11  31-Jul-11  31-Aug-11  30-Sep-11  31-Oct-11
OpenDate
2011-05-31  68.432797  81.696071  75.083249  66.659008  68.898034  72.622304
2011-06-30        NaN   1.711097   1.501082   1.625213   1.774645   1.661183
2011-07-31        NaN        NaN   0.422364   0.263561   0.203572   0.234376
2011-08-31        NaN        NaN        NaN   1.077009   1.226946   1.520701
2011-09-30        NaN        NaN        NaN        NaN   0.667091   0.495993

我想通过将第ith 行向左移动来转换它i-1：

            31-May-11  30-Jun-11  31-Jul-11  31-Aug-11  30-Sep-11  31-Oct-11
OpenDate
2011-05-31  68.432797  81.696071  75.083249  66.659008  68.898034  72.622304
2011-06-30  1.711097   1.501082   1.625213   1.774645   1.661183   NaN
2011-07-31  0.422364   0.263561   0.203572   0.234376   NaN        NaN
2011-08-31  1.077009   1.226946   1.520701   NaN        NaN        NaN
2011-09-30  0.667091   0.495993   NaN        NaN        NaN        NaN

编辑：

我不能排除矩阵的上部可能存在 NaN，所以我们可能会看到如下内容：

            31-May-11  30-Jun-11  31-Jul-11  31-Aug-11  30-Sep-11  31-Oct-11
OpenDate
2011-05-31  68.432797  81.696071  75.083249  66.659008  68.898034  72.622304
2011-06-30        NaN        NaN   1.501082   1.625213   1.774645   1.661183
2011-07-31        NaN        NaN   0.422364   0.263561   0.203572   0.234376
2011-08-31        NaN        NaN        NaN   1.077009   1.226946   1.520701
2011-09-30        NaN        NaN        NaN        NaN   0.667091   0.495993

应该变成

            31-May-11  30-Jun-11  31-Jul-11  31-Aug-11  30-Sep-11  31-Oct-11
OpenDate
2011-05-31  68.432797  81.696071  75.083249  66.659008  68.898034  72.622304
2011-06-30  NaN        1.501082   1.625213   1.774645   1.661183   NaN
2011-07-31  0.422364   0.263561   0.203572   0.234376   NaN        NaN
2011-08-31  1.077009   1.226946   1.520701   NaN        NaN        NaN
2011-09-30  0.667091   0.495993   NaN        NaN        NaN        NaN

任何想法如何实现这一目标？

谢谢，安妮

4

5 回答 5

4 
df.apply(lambda x: x.shift(-x.notnull().argmax()), 1)

lambda 函数找到第一个非空值的位置，并相应地移动行。这有两个问题：它没有利用已知结构（上三角形），因此可能会牺牲一些速度，此外，它可能会被数据中额外的 NaN 所欺骗。

更新

更强大的解决方案，使用 itertools 的计数器。

from itertools import count
c = count()
df.apply(lambda x: x.shift(-c.next() + 1), 1)

正如预期的那样，这要快一些。

In [47]: %timeit df.apply(lambda x: x.shift(-c.next() + 1), 1)
1000 loops, best of 3: 766 us per loop

In [49]: %timeit df.apply(lambda x: x.shift(-x.notnull().argmax()), 1)
1000 loops, best of 3: 1.08 ms per loop
于 2013-08-06T14:33:39.513 回答
4

这是您可以使用的一种方法numpy

输入：

In [96]: df
Out[96]:
                 1       2       3       4       5       6
0
2011-05-31  68.433  81.696  75.083  66.659  68.898  72.622
2011-06-30     NaN   1.711   1.501   1.625   1.775   1.661
2011-07-31     NaN     NaN   0.422   0.264   0.204   0.234
2011-08-31     NaN     NaN     NaN   1.077   1.227   1.521
2011-09-30     NaN     NaN     NaN     NaN   0.667   0.496

代码

roller = lambda (i, x): np.roll(x, -i)
row_terator = enumerate(df.values)
rolled = map(roller, row_terator)
result = DataFrame(np.vstack(rolled), index=df.index, columns=df.columns)

输出：

                 1       2       3       4       5       6
0
2011-05-31  68.433  81.696  75.083  66.659  68.898  72.622
2011-06-30   1.711   1.501   1.625   1.775   1.661     NaN
2011-07-31   0.422   0.264   0.204   0.234     NaN     NaN
2011-08-31   1.077   1.227   1.521     NaN     NaN     NaN
2011-09-30   0.667   0.496     NaN     NaN     NaN     NaN

让我们timeit

In [95]: %%timeit
   ....: roller = lambda (i, x): np.roll(x, -i)
   ....: row_terator = enumerate(df.values)
   ....: rolled = map(roller, row_terator)
   ....: result = DataFrame(np.vstack(rolled), index=df.index, columns=df.columns)
   ....:
10000 loops, best of 3: 101 us per loop

请注意，这np.roll是重要的事情。它需要一个数组、整数个要移位的位置和一个axis参数，这样您就可以ndarray沿其任何轴移动一个。

于 2013-08-06T15:32:43.627 回答
0

设置

In [23]: df = DataFrame(np.arange(40).reshape(10,4))

In [24]: df
Out[24]: 
    0   1   2   3
0   0   1   2   3
1   4   5   6   7
2   8   9  10  11
3  12  13  14  15
4  16  17  18  19
5  20  21  22  23
6  24  25  26  27
7  28  29  30  31
8  32  33  34  35
9  36  37  38  39

不知道这会有多快......

In [21]: def f(i,x):
   ....:     return x.shift(-i+1)
   ....: 

In [31]: DataFrame([ f(i,x) for i,x in df.iterrows() ])
Out[31]: 
    0   1   2   3
0 NaN   0   1   2
1   4   5   6   7
2   9  10  11 NaN
3  14  15 NaN NaN
4  19 NaN NaN NaN
5 NaN NaN NaN NaN
6 NaN NaN NaN NaN
7 NaN NaN NaN NaN
8 NaN NaN NaN NaN
9 NaN NaN NaN NaN
于 2013-08-06T15:12:48.323 回答
0

您可以计算 NaN 值，删除它们，然后在最后再次附加相同的数量。所以像：

def shift_df(row):

    n = len(row)

    new_row = row.dropna().tolist()
    new_row += ([np.nan]*(n-len(new_row)))

    return pd.Series(new_row, index=row.index)

df.apply(shift_df, axis=1)

你的数据框在哪里df。这仅在您的“正常”数据之间没有 NaN 值时才有效。

于 2013-08-06T14:24:39.480 回答
0

供将来参考，因为我必须在 numpy.

如果您将数据作为 numpy 数组，则另一种可能性如下：

In [75]: m
Out[75]:
array([[-0.69269313, -1.83256202, -0.61047484,  2.22505336,  0.65253538],
       [ 0.        ,  0.21960176,  1.82940845, -1.94429684, -0.42096599],
       [ 0.        ,  0.        ,  0.44483682, -0.56272361,  0.15877905],
       [ 0.        ,  0.        ,  0.        , -0.54694672,  0.20022243],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  1.82054127]])

In [76]: i = np.triu_indices(len(m))

In [77]: m2 = np.zeros_like(m)

In [78]: m2[i[0], i[1]-i[0]] = m[i]

In [79]: m2
Out[79]:
array([[-0.69269313, -1.83256202, -0.61047484,  2.22505336,  0.65253538],
       [ 0.21960176,  1.82940845, -1.94429684, -0.42096599,  0.        ],
       [ 0.44483682, -0.56272361,  0.15877905,  0.        ,  0.        ],
       [-0.54694672,  0.20022243,  0.        ,  0.        ,  0.        ],
       [ 1.82054127,  0.        ,  0.        ,  0.        ,  0.        ]])

当然，如果您想用 NaN 填充，您可以将m2矩阵初始化为该值而不是零。

不过，我不确定哪种方法更有效。

于 2020-03-17T18:18:08.810 回答