我下面的例程需要一个 urllib2.Requests 列表,并为每个请求生成一个新进程并触发它们。目的是为了异步速度,所以这一切都是一劳永逸的(不需要响应)。问题是下面代码中产生的进程永远不会终止。因此,在其中一些之后,盒子就会OOM。上下文:Django 网络应用程序。有什么帮助吗?
MP_CONCURRENT = int(multiprocessing.cpu_count()) * 2
if MP_CONCURRENT < 2: MP_CONCURRENT = 2
MPQ = multiprocessing.JoinableQueue(MP_CONCURRENT)
def request_manager(req_list):
try:
# put request list in the queue
for req in req_list:
MPQ.put(req)
# call processes on queue
worker = multiprocessing.Process(target=process_request, args=(MPQ,))
worker.daemon = True
worker.start()
# move on after queue is empty
MPQ.join()
except Exception, e:
logging.error(traceback.print_exc())
# prcoess requests in queue
def process_request(MPQ):
try:
while True:
req = MPQ.get()
dr = urllib2.urlopen(req)
MPQ.task_done()
except Exception, e:
logging.error(traceback.print_exc())