2

Given a matrix QT:

% ipython
Python 2.7.3
In [3]: QT.dtype
Out[3]: dtype('float64')

In [4]: QT.__class__
Out[4]: numpy.ndarray

In [5]: QT.flags
Out[5]:
      C_CONTIGUOUS : True
      F_CONTIGUOUS : False
      OWNDATA : True
      WRITEABLE : True
      ALIGNED : True
      UPDATEIFCOPY : False

I need the results of:

QT.T * QT

Problem: Whenever I try to compute these matrices multiplication, the memory overflows and the code stop running. This happen because of the matrix copy numpy is doing behind.

Tried solutions:

First:

Q = numpy.array(QT.T, order='C')
numpy.dot(Q, QT)

Second:

QT = numpy.array(QT, order='F')
Q = numpy.array(QT.T, order='F')
numpy.dot(Q, QT)

Third:

QT = numpy.matrix(QT)
QT = QT.copy('F')
Q = numpy.matrix(QT.T)
Q = Q.copy('F')
Q.dot(QT)

However, none of them is solving.

Question

How can I operate QT.T * QT without having the memory to explode?

References

http://numpy-discussion.10968.n7.nabble.com/inplace-matrix-multiplication-td21817.html

Is there an "enhanced" numpy/scipy dot method?

Numpy dot product very slow using ints

http://www.scipy.org/PerformanceTips

4

2 回答 2

2

你有没有尝试过：

shape = (QT.shape[2], QT.shape[2])
result = np.zeros(shape, dtype=QT.dtype)
np.dot(QT.T,  QT, out=result)

尝试运行上面的代码，看看哪一行（如果有的话）中断了。

于 2013-05-21T02:18:37.800 回答
1

如果结果不能全部放入核心内存，您可以将其放入内存映射数组中，以便溢出将写入您的硬盘：

shape = (QT.shape[2],)*2
result = np.memmap('result.dat', dtype=QT.dtype, mode='w+', shape=shape)
np.dot(QT.T, QT, out=result)

您可能还想看看这个在非常大的阵列上执行核外 SVD 的算法。

于 2014-01-07T19:43:13.083 回答