0

我想用 scipy 拟合许多发行版，并想为此使用某种多处理。像这样的东西：

import scipy.stats as ss
from pathos.multiprocessing import ProcessingPool
from multiprocessing import Pool

mp = Pool()
pp = ProcessingPool()

l = [0,1,2,3,4,6,7,8,9]
print map(ss.lognorm.fit,l) #method 0
print mp.map(ss.lognorm.fit,l) #method 1
print pp.map(ss.lognorm.fit,l) #method 2

方法 0 当然不是多处理，但有效。方法 1 和 2 都返回长回溯。有人有解决方法吗？

方法一错误：

Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
Process PoolWorker-2:
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
Process PoolWorker-4:
    return recv()
Traceback (most recent call last):
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-9:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))

方法二错误：

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/processing/pool.py", line 207, in _handleTasks
    put(task)
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 192, in dumps
    dump(obj, file, protocol, byref, fmode)#, strictio)
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 182, in dump
    pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 626, in save_function
    obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 826, in save_cell
    pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 794, in save_instancemethod0
    obj.im_class), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 658, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 794, in save_instancemethod0
    obj.im_class), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 615, in save_function
    if not _locate_function(obj): #, pickler._session):
  File "/usr/local/lib/python2.7/dist-packages/dill-0.2.2-py2.7.egg/dill/dill.py", line 604, in _locate_function
    found = _import_module(obj.__module__ + '.' + obj.__name__, safe=True)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
4

1 回答 1

2

方法 1 不起作用，因为您不能使用pickle. 方法 2 不起作用，因为正在做一些“棘手的scipy.stats”事情……作者（我）在没有先调查的情况下不太清楚它是什么。dillpathos

您可以看到问题不在于scipy.stats使用绑定方法（对于dillor而言不是问题pathos），而是它正在做一些重命名魔术......这就是为什么当您查看pathos调用的回溯时，您会看到_locate_function失败（它失败并发现None)... 这实际上就是方法 2 不起作用的原因。

>>> import scipy.stats as ss
>>>        
>>> ss.lognorm
<scipy.stats._continuous_distns.lognorm_gen object at 0x10932d6d0>

解决方法很简单。通过创建一个知道它在哪里的函数，可以更容易地找到该方法。

>>> import pathos.multiprocessing as mp
>>> p = mp.ProcessingPool()
>>>        
>>> def doit(x):
...   return ss.lognorm.fit(x)
... 
>>> p.map(doit, range(5))
[(1.0, 0.0, 1.0), (1.0, 1.0, 1.0), (1.0, 2.0, 1.0), (1.0, 3.0, 1.0), (1.0, 4.0, 1.0)]
于 2015-01-17T17:15:02.930 回答