rolling.corr
Pearson 是这样做的,因此您可以使用它。对于 Spearman,使用如下内容:
import pandas as pd
from numpy.lib.stride_tricks import as_strided
from numpy.lib import pad
import numpy as np
def rolling_spearman(seqa, seqb, window):
stridea = seqa.strides[0]
ssa = as_strided(seqa, shape=[len(seqa) - window + 1, window], strides=[stridea, stridea])
strideb = seqa.strides[0]
ssb = as_strided(seqb, shape=[len(seqb) - window + 1, window], strides =[strideb, strideb])
ar = pd.DataFrame(ssa)
br = pd.DataFrame(ssb)
ar = ar.rank(1)
br = br.rank(1)
corrs = ar.corrwith(br, 1)
return pad(corrs, (window - 1, 0), 'constant', constant_values=np.nan)
例如:
In [144]: df = pd.DataFrame(np.random.randint(0,1000,size=(10,2)), columns = list('ab'))
In [145]: df['corr'] = rolling_spearman(df.a, df.b, 4)
In [146]: df
Out[146]:
a b corr
0 429 922 NaN
1 618 382 NaN
2 476 517 NaN
3 267 536 -0.8
4 582 844 -0.4
5 254 895 -0.4
6 583 974 0.4
7 687 298 -0.4
8 697 447 -0.6
9 383 35 0.4
解释:numpy.lib.stride_tricks.as_strided
这是一种 hacky 方法,在这种情况下,它为我们提供了一个看起来像 2d 数组的序列视图,其中包含我们正在查看的序列的滚动窗口部分。
从此,简单了。Spearman相关相当于将序列转化为秩,取Pearson相关系数。有用的是,Pandas 快速实现了在DataFrame
s 上逐行执行此操作。然后在最后我们Series
用 NaN 值填充结果的开头(因此您可以将其作为列添加到您的数据框或其他任何内容)。
(个人注意:我花了很长时间试图弄清楚如何使用 numpy 和 scipy 有效地做到这一点,然后我才意识到你需要的一切都已经在 pandas 中了......!)。
为了展示这种方法相对于在滑动窗口上循环的速度优势,我制作了一个名为srsmall.py
包含的小文件:
import pandas as pd
from numpy.lib.stride_tricks import as_strided
import scipy.stats
from numpy.lib import pad
import numpy as np
def rolling_spearman_slow(seqa, seqb, window):
stridea = seqa.strides[0]
ssa = as_strided(seqa, shape=[len(seqa) - window + 1, window], strides=[stridea, stridea])
strideb = seqa.strides[0]
ssb = as_strided(seqb, shape=[len(seqb) - window + 1, window], strides =[strideb, strideb])
corrs = [scipy.stats.spearmanr(a, b)[0] for (a,b) in zip(ssa, ssb)]
return pad(corrs, (window - 1, 0), 'constant', constant_values=np.nan)
def rolling_spearman_quick(seqa, seqb, window):
stridea = seqa.strides[0]
ssa = as_strided(seqa, shape=[len(seqa) - window + 1, window], strides=[stridea, stridea])
strideb = seqa.strides[0]
ssb = as_strided(seqb, shape=[len(seqb) - window + 1, window], strides =[strideb, strideb])
ar = pd.DataFrame(ssa)
br = pd.DataFrame(ssb)
ar = ar.rank(1)
br = br.rank(1)
corrs = ar.corrwith(br, 1)
return pad(corrs, (window - 1, 0), 'constant', constant_values=np.nan)
并比较性能:
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: from srsmall import rolling_spearman_slow as slow
In [4]: from srsmall import rolling_spearman_quick as quick
In [5]: for i in range(6):
...: df = pd.DataFrame(np.random.randint(0,1000,size=(10*(10**i),2)), columns=list('ab'))
...: print len(df), " rows"
...: print "quick: ",
...: %timeit quick(df.a, df.b, 4)
...: print "slow: ",
...: %timeit slow(df.a, df.b, 4)
...:
10 rows
quick: 100 loops, best of 3: 3.52 ms per loop
slow: 100 loops, best of 3: 3.2 ms per loop
100 rows
quick: 100 loops, best of 3: 3.53 ms per loop
slow: 10 loops, best of 3: 42 ms per loop
1000 rows
quick: 100 loops, best of 3: 3.82 ms per loop
slow: 1 loop, best of 3: 430 ms per loop
10000 rows
quick: 100 loops, best of 3: 7.47 ms per loop
slow: 1 loop, best of 3: 4.33 s per loop
100000 rows
quick: 10 loops, best of 3: 50.2 ms per loop
slow: 1 loop, best of 3: 43.4 s per loop
1000000 rows
quick: 1 loop, best of 3: 404 ms per loop
slow:
在一百万行(在我的机器上)上,快速(熊猫)版本在不到半秒的时间内运行。上面没有显示,但 1000 万只用了 8.43 秒。缓慢的仍在运行,但假设线性增长继续下去,1M 需要大约 7 分钟,10M 需要一个多小时。