2

我对 Python 的多处理库有一个理解问题/问题:
为什么不同的进程(几乎)同时启动至少似乎是串行执行而不是并行执行?

任务是控制大量粒子的宇宙(一个粒子是一组 x/y/z 坐标和一个质量),并在利用多处理器环境的同时对它们执行各种分析。特别是对于下面显示的示例,我想计算所有粒子的质量中心。
因为该任务明确说要使用多个处理器,所以我没有使用线程库,因为有这个 GIL-thingy 将执行限制在一个处理器上。
这是我的代码:

from multiprocessing import Process, Lock, Array, Value
from random import random
import math
from time import time

def exercise2(noOfParticles, noOfProcs):
    startingTime = time()
    particles = []
    processes = []
    centerCoords = Array('d',[0,0,0])
    totalMass = Value('d',0)
    lock = Lock()

    #create all particles
    for i in range(noOfParticles):
        p = Particle()
        particles.append(p)

    for i in range(noOfProcs):
        #determine the number of particles every process needs to analyse
        particlesPerProcess = math.ceil(noOfParticles / noOfProcs)
        #create noOfProcs Processes, each with a different set of particles        
        p = Process(target=processBatch, args=(
            particles[i*particlesPerProcess:(i+1)*particlesPerProcess],
            centerCoords, #handle to shared memory
            totalMass, #handle to shared memory
            lock, #handle to lock
            'batch'+str(i)), #also pass name of process for easier logging
            name='batch'+str(i))
        processes.append(p)
        print('created proc:',i)

    #start all processes
    for p in processes:
        p.start() #here, the program waits for the started process to terminate. why?

    #wait for all processes to finish
    for p in processes:
        p.join()

    #normalize the coordinates
    centerCoords[0] /= totalMass.value
    centerCoords[1] /= totalMass.value
    centerCoords[2] /= totalMass.value

    print(centerCoords[:])
    print('total time used', time() - startingTime, ' seconds')


class Particle():
    """a particle is a very simple physical object, having a set of x/y/z coordinates and a mass.
    All values are randomly set at initialization of the object"""

    def __init__(self):
        self.x = random() * 1000
        self.y = random() * 1000
        self.z = random() * 1000
        self.m = random() * 10

    def printProperties(self):
        attrs = vars(self)
        print ('\n'.join("%s: %s" % item for item in attrs.items()))

def processBatch(particles,centerCoords,totalMass,lock,name):
    """calculates the mass-weighted sum of all coordinates of all particles as well as the sum of all masses.
    Writes the results into the shared memory centerCoords and totalMass, using lock"""

    print(name,' started')
    mass = 0
    centerX = 0
    centerY = 0
    centerZ = 0

    for p in particles:
        centerX += p.m*p.x
        centerY += p.m*p.y
        centerZ += p.m*p.z
        mass += p.m

    with lock:
        centerCoords[0] += centerX
        centerCoords[1] += centerY
        centerCoords[2] += centerZ
        totalMass.value += mass

    print(name,' ended')

if __name__ == '__main__':
    exercise2(2**16,6)

现在我希望所有进程大约在同一时间启动并并行执行。但是当我查看程序的输出时,这看起来好像进程正在串行执行:

created proc: 0
created proc: 1
created proc: 2
created proc: 3
created proc: 4
created proc: 5
batch0  started
batch0  ended
batch1  started
batch1  ended
batch2  started
batch2  ended
batch3  started
batch3  ended
batch4  started
batch4  ended
batch5  started
batch5  ended
[499.72234074100135, 497.26586187539453, 498.9208784328791]
total time used 4.7220001220703125  seconds

此外,当使用 Eclipse 调试器单步执行程序时,我可以看到程序总是等待一个进程终止,然后在标有以“为什么?”结尾的注释的行开始下一个进程。当然,这可能只是调试器,但是当我查看正常运行中产生的输出时,这正是上图所示。

  • 这些进程是否并行执行而我只是因为标准输出的一些共享问题而看不到它?
  • 如果进程是串行执行的:为什么?我怎样才能让它们并行运行?

非常感谢您对理解这一点的任何帮助。

我在具有双核 Intel 处理器的 Windows 7 机器上使用 Python 3.2.3 从 PyDev 和命令行执行了上述代码。


编辑:
由于程序的输出,我误解了这个问题:进程实际上是并行运行的,但是腌制大量数据并将其发送到子进程的开销需要很长时间,以至于完全扭曲了画面。
将粒子(即数据)的创建移至子进程,以便它们不必首先被腌制,从而消除了所有问题,并导致程序的有用的并行执行。
因此,为了解决这个任务,我必须将粒子保存在共享内存中,这样它们就不必传递给子进程。

4

1 回答 1

2

我在我的系统(Python 2.6.5)上运行了您的代码,它几乎立即返回结果,这让我认为您的任务规模可能太小,以至于进程在下一个开始之前完成(请注意,启动进程较慢比旋转线程)。我质疑total time used 4.7220001220703125 seconds您的结果,因为这比我的系统运行相同代码所花费的时间长约 40 倍。我将粒子数放大到2**20,得到以下结果:

('created proc:', 0)
('created proc:', 1)
('created proc:', 2)
('created proc:', 3)
('created proc:', 4)
('created proc:', 5)
('batch0', ' started')
('batch1', ' started')
('batch2', ' started')
('batch3', ' started')
('batch4', ' started')
('batch5', ' started')
('batch0', ' ended')
('batch1', ' ended')
('batch2', ' ended')
('batch3', ' ended')
('batch5', ' ended')
('batch4', ' ended')
[500.12090773656854, 499.92759577086059, 499.97075039983588]
('total time used', 5.1031057834625244, ' seconds')

这更符合我的预期。如果你增加任务大小,你会得到什么?

于 2012-06-04T18:53:39.653 回答