0

我有以下python代码：

H1 = [[0.04,0.03,0.01,0.002],[0.02,0.04,0.001,0.5]]
H2 = [[0.06,0.02,0.02,0.004],[0.8,0.09,0.6,0.1]]    

D1 = [0.01,0.02,0.1,0.01]    
D2 = [0.1,0.3,0.01,0.4] 

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T = []    
append2 = T.append   
E = []    
append3 = E.append   

for h1,h2 in itertools.izip(H1,H2) 
    Err = []    
    append1 = Err.append
    for v in h1:    

        L1 = [1 if i>=v else 0 for i in h1]    
        L2 = [1 if i>=v else 0 for i in h2]    

        Sp = np.dot(D1,L1)     
        Sn = np.dot(D2,L2)    

        err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
        append1(err)  

    b = np.argmin(Err)    
    append2(h1[b])    
    append3(Err[b])

这只是一个示例代码。我需要运行内部 for 循环大约 20,000 次（这里它只运行两次）。但是内部 for 循环需要很长时间，因此使用起来不切实际。在 line profiler 中，它显示 line Sp = np.dot(D1,L1)，Sn = np.dot(D2,L2)并且 b = np.argmin(Err) 是最耗时的。如何减少上述代码所花费的时间。

任何帮助都感激不尽。

谢谢！

4

4 回答 4

4

如果您使用带有 numpy 数组而不是列表的 numpy 函数，您可以获得相当大的速度提升。大多数 numpy 函数会在内部将列表转换为数组，这会增加运行时的大量开销。这是一个简单的例子：

In [16]: a = range(10)

In [17]: b = range(10)

In [18]: aa = np.array(a)

In [19]: bb = np.array(b)

In [20]: %timeit np.dot(a, b)
10000 loops, best of 3: 54 us per loop

In [21]: %timeit np.dot(aa, bb)
100000 loops, best of 3: 3.4 us per loop

numpy.dot在这种情况下，使用数组调用时运行速度快 16 倍。此外，当您使用 numpy 数组时，您将能够简化一些代码，这也应该有助于它运行得更快。例如 ifh1是一个数组，L1 = [1 if i>=v else 0 for i in h1]可以写成h1 > vwhich 返回一个数组并且应该运行得更快。贝娄我已经继续并用数组替换了您的列表，这样您就可以看到它的样子。

import numpy as np

H1 = np.array([[0.04,0.03,0.01,0.002],[0.02,0.04,0.001,0.5]])
H2 = np.array([[0.06,0.02,0.02,0.004],[0.8,0.09,0.6,0.1]])

D1 = np.array([0.01,0.02,0.1,0.01])
D2 = np.array([0.1,0.3,0.01,0.4])

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T = np.zeros(H1.shape[0])
E = np.zeros(H1.shape[0])

for i in range(len(H1)):
    h1 = H1[i]
    h2 = H2[i]
    Err = np.zeros(len(h1))

    for j in range(len(h1)):    
        v = h1[j]

        L1 = h1 > v
        L2 = h2 > v

        Sp = np.dot(D1, L1)     
        Sn = np.dot(D2, L2)    

        err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
        Err[j] = err

    b = np.argmin(Err)
    T[i] = h1[b]
    E[i] = Err[b]

一旦您对 numpy 数组更加熟悉，您可能希望考虑使用broadcasting来表达至少您的内部循环。对于某些应用程序，使用广播可能比 python 循环更有效。祝你好运，希望有帮助。

于 2013-10-13T16:41:06.617 回答
2

您需要将数据保存在 ndarray 类型中。当您对列表执行 numpy 操作时，它每次都必须构造一个新数组。我修改了您的代码以运行可变次数，并且发现它对于 10000 次迭代也是 ~1s。将数据类型更改为 ndarrays 将其减少了大约两倍，我认为仍有一些改进（第一个版本有一个错误，使其执行速度过快）

import itertools
import numpy as np
N = 10000
H1 = [np.array([0.04,0.03,0.01,0.002])] * N
H2 = [np.array([0.06,0.02,0.02,0.004])] * N

D1 = np.array([0.01,0.02,0.1,0.01]    )
D2 = np.array([0.1,0.3,0.01,0.4] )

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T = []    
append2 = T.append   
E = []    
append3 = E.append   

for h1,h2 in itertools.izip(H1,H2):
    Err = []    
    append1 = Err.append
    for v in h1:

        #L1 = [1 if i>=v else 0 for i in h1]    
        #L2 = [1 if i>=v else 0 for i in h2]    
        L1 = h1 > v
        L2 = h2 > v
        Sp = np.dot(D1,L1)     
        Sn = np.dot(D2,L2)    

        err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
        append1(err)  

    b = np.argmin(Err)    
    append2(h1[b])    
    append3(Err[b])
于 2013-10-13T16:50:43.003 回答
1

您的列表理解中有一些容易实现的目标：

L1 = [1 if i>=v else 0 for i in h1]
L2 = [1 if i>=v else 0 for i in h2]

上面可以写成：

L1 = [i>=v for i in h1]
L2 = [i>=v for i in h2]

因为布尔值是整数的子类，True而且False已经是 1 和 0，只是穿着花哨的衣服。

err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
append1(err)

您可以结合以上两行来避免变量分配和访问。

如果将代码放在函数中，所有局部变量的使用都会稍微快一些。此外，您使用的任何全局函数或方法（例如min, np.dot）都可以使用默认参数在函数签名中转换为局部变量。np.dot是一个特别慢的调用（除了操作本身需要多长时间），因为它涉及属性查找。这类似于您已经使用列表append方法进行的优化。

现在我想这一切都不会真正影响性能，因为你的问题似乎真的是“我怎样才能让 NumPy 更快？” （其他人对你来说是最重要的）但它们可能会产生一些影响并且值得去做。

于 2013-10-13T15:47:11.603 回答
0

如果我正确理解了np.dot()两个维度 1 列表上的指令是什么，那么在我看来，以下代码应该与您的代码相同。
请你测试一下它的速度好吗？

它的原理是使用索引而不是列表的元素，并使用定义为函数默认值的列表的特性

H1 = [[0.04,0.03,0.01,0.002],[0.02,0.04,0.001,0.5]]
H2 = [[0.06,0.02,0.02,0.004],[0.8,0.09,0.6,0.1]]    

D1 = [0.01,0.02,0.1,0.01]    
D2 = [0.1,0.3,0.01,0.4] 

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T,E = [],[]    
append2 = T.append      
append3 = E.append 

ONE,TWO = [],[]

def zoui(v, ONE=ONE,TWO=TWO,
         D1=D1,D2=D2,Tp=Tp,Tn=Tn,tu0123 = (0,1,2,3)):
    diff =  sum(D1[i] if ONE[i]>=v else 0 for i in tu0123)\
           -sum(D2[i] if TWO[i]>=v else 0 for i in tu0123)
    #or maybe
    #diff =  sum(D1[i] * ONE[i]>=v for i in tu0123)\
    #       -sum(D2[i] * TWO[i]>=v for i in tu0123)

    return min(Tn+diff,Tp-diff)

for n in xrange(len(H1)):
    ONE[:] = H1[n]
    TWO[:] = H2[n]
    Err = map(zoui,ONE)
    b = np.argmin(Err)    
    append2(ONE[b])    
    append3(Err[b])
于 2013-10-13T16:46:49.453 回答