python - Efficient way to normalize a Scipy Sparse Matrix

Question

I'd like to write a function that normalizes the rows of a large sparse matrix (such that they sum to one).

from pylab import *
import scipy.sparse as sp

def normalize(W):
    z = W.sum(0)
    z[z < 1e-6] = 1e-6
    return W / z[None,:]

w = (rand(10,10)<0.1)*rand(10,10)
w = sp.csr_matrix(w)
w = normalize(w)

However this gives the following exception:

File "/usr/lib/python2.6/dist-packages/scipy/sparse/base.py", line 325, in __div__
     return self.__truediv__(other)
File "/usr/lib/python2.6/dist-packages/scipy/sparse/compressed.py", line 230, in  __truediv__
   raise NotImplementedError

Are there any reasonably simple solutions? I have looked at this, but am still unclear on how to actually do the division.

score 46 · Accepted Answer

这已在scikit-learn sklearn.preprocessing.normalize中实现。

from sklearn.preprocessing import normalize
w_normalized = normalize(w, norm='l1', axis=1)

axis=1应该按行axis=0归一化，按列归一化。使用可选参数copy=False来修改矩阵。

score 3 · Accepted Answer

虽然 Aarons 的回答是正确的，但当我想对绝对值的最大值进行归一化时，我实现了一个解决方案，而 sklearn 没有提供。我的方法使用非零条目并在 csr_matrix.data 数组中找到它们以快速替换那里的值。

def normalize_sparse(csr_matrix):
    nonzero_rows = csr_matrix.nonzero()[0]
    for idx in np.unique(nonzero_rows):
        data_idx = np.where(nonzero_rows==idx)[0]
        abs_max = np.max(np.abs(csr_matrix.data[data_idx]))
        if abs_max != 0:
            csr_matrix.data[data_idx] = 1./abs_max * csr_matrix.data[data_idx]

与 sunan 的解决方案相比，这种方法不需要将矩阵转换为密集格式（这可能会引发内存问题），也不需要矩阵乘法。我在一个稀疏的形状矩阵 (35'000, 486'000) 上测试了该方法，大约需要 18 秒。

score 2 · Accepted Answer

这是我的解决方案。

转置A
计算每列的总和
用总和的倒数格式化对角矩阵 B
A*B 等于归一化

转置 C

import scipy.sparse as sp
import numpy as np
import math

minf = 0.0001

A = sp.lil_matrix((5,5))
b = np.arange(0,5)
A.setdiag(b[:-1], k=1)
A.setdiag(b)
print A.todense()
A = A.T
print A.todense()

sum_of_col = A.sum(0).tolist()
print sum_of_col
c = []
for i in sum_of_col:
    for j in i:
        if math.fabs(j)<minf:
            c.append(0)
        else:
            c.append(1/j)

print c

B = sp.lil_matrix((5,5))
B.setdiag(c)
print B.todense()

C = A*B
print C.todense()
C = C.T
print C.todense()

score 1 · Accepted Answer

我发现这是一种不使用内置函数的优雅方式。

import scipy.sparse as sp

def normalize(W):
    #Find the row scalars as a Matrix_(n,1)
    rowSumW = sp.csr_matrix(W.sum(axis=1))
    rowSumW.data = 1/rowSumW.data

    #Find the diagonal matrix to scale the rows
    rowSumW = rowSumW.transpose()
    scaling_matrix = sp.diags(rowSumW.toarray()[0])

    return scaling_matrix.dot(W)

score 0 · Accepted Answer

在不导入 sklearn 的情况下，转换为密集或乘法矩阵并利用 csr 矩阵的数据表示：

from scipy.sparse import isspmatrix_csr

def normalize(W):
    """ row normalize scipy sparse csr matrices inplace.
    """
    if not isspmatrix_csr(W):
        raise ValueError('W must be in CSR format.')
    else:
        for i in range(W.shape[0]):
            row_sum = W.data[W.indptr[i]:W.indptr[i+1]].sum()
            if row_sum != 0:
                W.data[W.indptr[i]:W.indptr[i+1]] /= row_sum

请记住，这W.indices是列索引 W.data的数组，是对应的非零值的数组，并且W.indptr指向索引和数据中开始的行。

numpy.abs()如果您需要 L1 范数或用于numpy.max()按每行的最大值进行归一化，您可以在求和时添加 a 。

python - Efficient way to normalize a Scipy Sparse Matrix

5 回答 5

Related

Reference