0

我正在尝试并行化一些动态 LCA（就像 10 ......而我应该运行超过 100 ......），我得到了这个

Can't expand MemType 1: jcol 19852
Can't expand MemType 0: jcol 17930
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "multiprocess_both.py", line 129, in dyn_lca
    timeline=dynamic_lca.calculate()
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2temporalis/dynamic_lca.py", line 54, in calculate
    **self.gt_kwargs
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2calc/graph_traversal.py", line 44, in calculate
    lca, supply, score = self.build_lca(demand, method)
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2calc/graph_traversal.py", line 107, in build_lca
    lca.decompose_technosphere()
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/bw2calc/lca.py", line 216, in decompose_technosphere
    self.solver = factorized(self.technosphere_matrix.tocsc())
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 377, in factorized
    return splu(A).solve
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/site-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 257, in splu
    ilu=False, options=_options)
MemoryError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "multiprocess_both.py", line 181, in <module>
    dyn_multi=[x for x in pool_dyn_multi.get()] 
  File "/home/giuseppec/miniconda3/envs/bw2/lib/python3.4/multiprocessing/pool.py", line 608, in get
    raise self._value
MemoryError

似乎 scipy 在矩阵求逆过程中内存不足（我当然不是最了解这个问题，但我搜索了一下）我只是尝试安装 scikit-umfpack 以查看是否有变化......它正在运行但似乎仍然非常缓慢而且不是很有希望，而且只是返回这个警告：

Warning: (almost) singular matrix! (estimated cond. number: 3.33e+14)

实际上不知道如何处理这个......有人有吗？

PS：我在这里发帖是因为它在我看来不是 bw2 的错误，如果是这样会打开一个问题

4

1 回答 1

0

你的游泳池大小是多少？可能最容易尝试的方法是减少并行运行的作业数量，但将 cpu 的数量设置为可用数量的一半。

也有可能某处存在内存泄漏，并且在作业完成后内存没有被释放。您可以通过在构建 MP Poolmaxtasksperchild=1时进行设置来回避这一点。

于 2016-04-23T21:18:27.163 回答