我有一个非常稀疏的矩阵，比如 5000x3000，双精度浮点数。这个矩阵的 80% 是零。我需要计算每一行的总和。所有这些都在 python/cython 中。我想加快这个过程。因为我需要计算这个总和数百万次，所以我认为如果我制作非零元素的索引并仅对它们求和，它会更快。结果变得比所有零的原始“蛮力”总和慢得多。
这是一个最小的例子：
#cython: language_level=2
import numpy as np
cimport numpy as np
import time
cdef int Ncells = 5000, KCells = 400, Ne= 350
cdef double x0=0.1, x1=20., x2=1.4, x3=2.8, p=0.2
# Setting up weight
all_weights = np.zeros( (Ncells,KCells) )
all_weights[ :Ne, :Ne ] = x0
all_weights[ :Ne, Ne: ] = x1
all_weights[Ne: , :Ne ] = x2
all_weights[Ne: , Ne: ] = x3
all_weights = all_weights * (np.random.rand(Ncells,KCells) < p)
# Making a memory view
cdef np.float64_t[:,:] my_weights = all_weights
# make an index of non zero weights
x,y = np.where( np.array(my_weights) > 0.)
#np_pawid = np.column_stack( (x ,y ) )
np_pawid = np.column_stack( (x ,y ) ).astype(int)
cdef np.int_t[:,:] pawid = np_pawid
# Making vector for column sum
summEE = np.zeros(KCells)
# Memory view
cdef np.float64_t [:] my_summEE = summEE
cdef int cc,dd,i
# brute-force summing
ntm = time.time()
for cc in range(KCells):
my_summEE[cc] = 0
for dd in range(Ncells):
my_summEE[cc] += my_weights[dd,cc]
stm = time.time()
print "BRUTE-FORCE summation : %f s"%(stm-ntm)
my_summEE[:] = 0
# summing only non zero indices
ntm = time.time()
for dd,cc in pawid:
my_summEE[cc] += my_weights[dd,cc]
stm = time.time()
print "INDEX summation : %f s"%(stm-ntm)
my_summEE[:] = 0
# summing only non zero indices unpacked by zip
ntm = time.time()
for dd,cc in zip(pawid[:,0],pawid[:,1]):
my_summEE[cc] += my_weights[dd,cc]
stm = time.time()
print "ZIPPED INDEX summation : %f s"%(stm-ntm)
my_summEE[:] = 0
# summing only non zero indices unpacked by zip
ntm = time.time()
for i in range(pawid.shape[0]):
dd = pawid[i,0]
cc = pawid[i,1]
my_summEE[cc] += my_weights[dd,cc]
stm = time.time()
print "INDEXING over INDEX summation: %f s"%(stm-ntm)
# Numpy brute-froce summing
ntm = time.time()
sumwee = np.sum(all_weights,axis=0)
stm = time.time()
print "NUMPY BRUTE-FORCE summation : %f s"%(stm-ntm)
#>
print
print "Number of brute-froce summs :",my_weights.shape[0]*my_weights.shape[1]
print "Number of indexing summs :",pawid.shape[0]
#<
我在 Raspberry Pi 3 上运行它，但在 PC 上似乎也有相同的结果。
BRUTE-FORCE summation : 0.381014 s
INDEX summation : 18.479018 s
ZIPPED INDEX summation : 3.615952 s
INDEXING over INDEX summation: 0.450131 s
NUMPY BRUTE-FORCE summation : 0.013017 s
Number of brute-froce summs : 2000000
Number of indexing summs : 400820
NUMPY BRUTE-FORCE in Python : 0.029143 s
谁能解释为什么 cython 代码比 numpy 慢 3-4 倍？为什么索引，它将总和的数量从 2000000 减少到 400820，慢了 45 倍？这没有任何意义。