5

I am using a Pool to benefit of multiple cores. Each worker in the pool needs its own Calculator object. The initialization of calculator is quite time consuming, so I would like to have it generated only once per worker in the pool and not every time, a new task arrives. The only way, I got this working was by using the “ugly“ keyword global. Is there a “cleaner” way to implement this?

I would like to avoid queues (parent thread is often sigkill’d and leaves child processes when using queues) and managers (performance too slow).

#!/usr/bin/python
# -*- coding: utf-8 -*-

import multiprocessing

def init_pool():
    global calculator
    calculator = Calculator()   # should only executed ones per worker

def run_pool(args):
    return calculator.calculate(*args)  # time consuming calculation

class Organiser():
    def __init__(self):
        self.__pool = multiprocessing.Pool(initializer=init_pool)

    def process(self, tasks):
        results = self.__pool.map(run_pool, tasks)
        return results
4

1 回答 1

2

我没有看到一种方法来实现你想要的（每个工人只初始化一次）。

但是，如果您想为整个工作组只初始化一次“计算器”，则以下方法似乎有效。

def run_pool(args):
    calculator,arg = args
    return calculator.calculate(arg)  # time consuming calculation

class Organiser():
    def __init__(self):
        self.calculator = Calculator()
        self.__pool = multiprocessing.Pool(processes=4)

    def process(self, tasks):
        results = self.__pool.map(run_pool, [(self.calculator,data) for data in tasks])
        return results

要为每个工作人员仅初始化一次，在我看来，您必须使用全局变量或单例（等效）。我也将等待您的问题的其他答案:)

问候， 悉达多

于 2012-10-31T06:45:43.823 回答