1

Is there a way to perform an operation between two dataframes?

For example, suppose we have a couple of dataframes:

df1 = pd.DataFrame({'a': randn(5), 'b': randn(5)})
df2 = pd.DataFrame({'c': randn(5), 'd': randn(5)})

df1
          a         b
0 -0.287740 -0.255126
1 -0.356745  0.632524
2 -0.379608 -0.876348
3 -0.596401  0.937805
4  0.969356  0.421352

df2
          c         d
0 -0.505406 -0.921449
1  0.508703  0.844641
2  0.300125 -0.942838
3  0.711138 -0.364033
4 -0.370174 -1.284353

And I'd like to perform an operation (maybe, a correlation) between the rows of df1 and df2. Using this example, I want something like this in a new dataframe:

corr([-0.287740 -0.255126],[-0.505406 -0.921449])
corr([-0.356745  0.632524],[0.508703  0.844641])
...
corr([0.969356  0.421352],[-0.370174 -1.284353])

Although, using the respective columns would also be possible by simply transposing df1 and df2.

I think this should be possible, although the original problem was related to numpy arrays, which would be stated in the following way:

Suppose a couple of arrays i and j:

i = np.random.random((5,2))
j = np.random.random((5,2))

i
array([[ 0.88005044,  0.60633474],
       [ 0.1183816 ,  0.61705454],
       [ 0.24259704,  0.8490759 ],
       [ 0.66581795,  0.31805491],
       [ 0.89337085,  0.21212527]])

j
array([[ 0.52745975,  0.07748147],
       [ 0.36152729,  0.74438265],
       [ 0.48207699,  0.28462384],
       [ 0.08623375,  0.55043213],
       [ 0.26371755,  0.23409753]])

How can I correlate the first array of i [ 0.88005044, 0.60633474] with the first array of j [ 0.52745975, 0.07748147]?

4

2 回答 2

2

对于 numpy 的事情（这是你原来的问题）不应该只是：

map(np.correlate, i, j)

解决这个问题？

>>> i = np.random.random((3,2))
>>> j = np.random.random((3,2))
>>> i
array([[ 0.19553363,  0.88041704],
       [ 0.15929013,  0.95699672],
       [ 0.83473442,  0.9943525 ]])
>>> j
array([[ 0.25712069,  0.37596445],
       [ 0.35283409,  0.73069007],
       [ 0.48959406,  0.17451211]])
>>> map(np.correlate, i, j)
[array([ 0.38128125]),
 array([ 0.75547099]),
 array([ 0.58220756])]

更新：感谢@PhillipCloud 提供更短的版本提示。

于 2013-08-17T23:40:25.610 回答
1

你可以做

df1 = DataFrame(randn(10, 2))
df2 = DataFrame(randn(10, 2))
result = DataFrame(empty((2, 2)))
for (i, coli) in df1.iteritems():
    for (j, colj) in df2.iteritems():
        result.iloc[i, j] = coli.corr(colj)
result

输出：

       0      1
0 -0.071  0.726
1  0.316 -0.277

n * m这将随位置缩放，n == df1.shape[1]因此m == df2.shape[1]您应该转置任何一个的轴（您在上面说过您可以选择这样做），以便最小的轴是列轴。

更新：我完全忘记了DataFrame.corrwith()！

In [12]:

df1 = DataFrame(rand(100000, 20))
df2 = DataFrame(rand(100000, 20))
c1 = df1.corrwith(df2)
c1

Out[12]:
0    -0.003
1     0.000
2     0.001
3    -0.005
4     0.002
5     0.000
6    -0.001
7     0.003
8     0.001
9     0.007
10   -0.003
11    0.003
12    0.006
13    0.003
14   -0.004
15    0.001
16   -0.000
17    0.001
18    0.003
19   -0.001
dtype: float64

In [13]:

timeit df1.corrwith(df2)
10 loops, best of 3: 154 ms per loop

In [14]:

a, b = df1.values.T, df2.values.T
c2 = map(correlate, a, b)
c2

Out[14]:
[array([ 24903.2259]),
 array([ 25006.8102]),
 array([ 25020.8546]),
 array([ 24929.0867]),
 array([ 24991.4906]),
 array([ 25040.7961]),
 array([ 24916.7744]),
 array([ 25073.9599]),
 array([ 25080.4035]),
 array([ 25102.4981]),
 array([ 24894.667]),
 array([ 25071.7446]),
 array([ 25068.3474]),
 array([ 25104.1667]),
 array([ 24904.1676]),
 array([ 25115.5725]),
 array([ 25019.9898]),
 array([ 25055.4906]),
 array([ 24957.8537]),
 array([ 25054.7859])]

In [15]:

timeit map(correlate, a, b)
10 loops, best of 3: 27.9 ms per loop

In [16]:

df1_nans = df1[df1 > 0.5]
df2_nans = df2[df2 > 0.5]
c1_nans = df1_nans.corrwith(df2_nans)

In [17]:

timeit df1_nans.corrwith(df2_nans)
10 loops, best of 3: 163 ms per loop

In [18]:

a_nans, b_nans = df1_nans.values.T, df2_nans.values.T
map(correlate, a_nans, b_nans)

Out[18]:
[array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan]),
 array([ nan])]

所以，回顾一下，correlate（除了给出不同的结果，因为它是一个不同的函数）在具有nan值的对象上完全失败。corrwith另一方面，不会失败（并且启动起来更灵活），但速度要慢 3 倍。

于 2013-08-17T23:36:44.157 回答