4

我正在使用此代码来抓取 API：

submissions = get_submissions(1)
with futures.ProcessPoolExecutor(max_workers=4) as executor:
#or using this: with futures.ThreadPoolExecutor(max_workers=4) as executor:
    for s in executor.map(map_func, submissions):
        collection_front.update({"time_recorded":time_recorded}, {'$push':{"thread_list":s}}, upsert=True)

它与线程一起工作得很好/很快，但是当我尝试使用进程时，我得到一个完整的队列并且这个错误：

  File "/usr/local/lib/python3.4/dist-packages/praw/objects.py", line 82, in __getattr__
    if not self.has_fetched:
RuntimeError: maximum recursion depth exceeded
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/concurrent/futures/process.py", line 251, in _queue_management_worker
    shutdown_worker()
  File "/usr/lib/python3.4/concurrent/futures/process.py", line 209, in shutdown_worker
    call_queue.put_nowait(None)
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 131, in put_nowait
    return self.put(obj, False)
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 82, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "reddit_proceses.py", line 64, in <module>
    for s in executor.map(map_func, submissions):
  File "/usr/lib/python3.4/concurrent/futures/_base.py", line 549, in result_iterator
    yield future.result()
  File "/usr/lib/python3.4/concurrent/futures/_base.py", line 402, in result
    return self.__get_result()
  File "/usr/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

请注意，最初这些过程对于小型数据检索工作得很好而且非常快，但现在它们根本不起作用。这是一个错误还是发生了什么，PRAW 对象会导致进程出现递归错误，但线程不会出现递归错误？

4

1 回答 1

2

我在从线程转移到进程时遇到了类似的问题，只有我使用的是 executor.submit。我认为这可能与您遇到的问题相同，但我不能确定，因为我不知道您的代码在什么上下文中运行。

就我而言，发生的事情是：我将我的代码作为脚本运行，并且我没有使用总是推荐的if __name__ == "__main__":. 看起来当使用执行器运行一个新进程时，python 会加载 .py 文件并运行提交中指定的函数。因为它加载了文件，所以存在于主文件中的代码（不在函数内部或上面的 if 语句中）被运行，因此每个进程将再次运行一个新进程，具有无限递归。

看起来线程不会发生这种情况。

于 2015-10-01T21:53:28.750 回答