0

我在我的程序中使用多处理模块进行并行处理，我想获取多进程之间的共享dict对象，我可以在多进程正常关闭时进行，但按CTRL + C时无法获取，我怎样才能实现我的目标？我的代码如下

#!/usr/bin/python
from multiprocessing import Process, Manager, Pool
import os
import signal
import time

def init_worker():
    signal.signal(signal.SIGINT, signal.SIG_IGN)

def run_worker(i,my_d):
    print 'Work Started: %d %d' % (os.getpid(), i)
    for j in range(5):
        print j
        tmp1 = str(i) + str(j)
        my_d[tmp1] = j
        time.sleep(1)

def main():
    print "Initializng 3 workers"
    pool = Pool(3, init_worker)

    manager = Manager()
    my_d = manager.dict()
    try:
        for i in range(3):
            pool.apply_async(run_worker,args=(i,my_d))
        pool.close()
        pool.join()
        print my_d
# When process is closed normally, I could get the my_d successfully

    except KeyboardInterrupt:
        print "Caught KeyboardInterrupt, terminating workers"
        pool.terminate()
        pool.join()
        print my_d
#When process is closed by Ctrl+C, couldn't I get the my_d ?

if __name__ == "__main__":
    main()
4

2 回答 2

0

您需要一个共享字典，来自多处理中的Manager -Object。

在这里看到这个类似的问题（第一个答案）： Python multiprocessing: How do I share a dict between multiple processes？

于 2012-09-16T11:38:48.557 回答
0

查看中断父进程时产生的错误：

Caught KeyboardInterrupt, terminating workers
<DictProxy object, typeid 'dict' at 0x801abe150; '__str__()' failed>

尝试更改print "Caught KeyboardInterrupt, terminating workers"为print len(my_d)，您可以详细了解会发生什么。请注意，这是在您尝试终止/加入工人池之前：

Traceback (most recent call last):
  File "manager-test.py", line 39, in <module>
    main()
  File "manager-test.py", line 33, in main
    print len(my_d)
  File "<string>", line 2, in __len__
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
    self._connect()
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/local/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/local/lib/python2.7/multiprocessing/connection.py", line 293, in SocketClient
    s.connect(address)
  File "/usr/local/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory

当您中断主程序时，从子进程到管理器的连接将中断。这会使管理器（及其管理的对象）处于不可用状态。从管理器到子进程的套接字连接不再起作用，因此代理无法获取数据。

如果你想中断长时间运行的进程而不丢失数据，你应该更温和地做，我认为。像这样的东西：

import select
import sys

print 'Type q<enter> it you want to quit...'
while True:
    r, foo, bla = select.select([sys.stdin], [], [], 1)
    if len(r):
        what = sys.stdin.readline()
        if 'q' in what:
            print 'bye!'
            break;
    # E.g. check on the progress of your calculation here
# Close and join the pool here, and do other clean-up.
于 2012-09-16T12:44:26.240 回答