我对 Python 的多处理库有一个理解问题/问题:
为什么不同的进程(几乎)同时启动至少似乎是串行执行而不是并行执行?
任务是控制大量粒子的宇宙(一个粒子是一组 x/y/z 坐标和一个质量),并在利用多处理器环境的同时对它们执行各种分析。特别是对于下面显示的示例,我想计算所有粒子的质量中心。
因为该任务明确说要使用多个处理器,所以我没有使用线程库,因为有这个 GIL-thingy 将执行限制在一个处理器上。
这是我的代码:
from multiprocessing import Process, Lock, Array, Value
from random import random
import math
from time import time
def exercise2(noOfParticles, noOfProcs):
startingTime = time()
particles = []
processes = []
centerCoords = Array('d',[0,0,0])
totalMass = Value('d',0)
lock = Lock()
#create all particles
for i in range(noOfParticles):
p = Particle()
particles.append(p)
for i in range(noOfProcs):
#determine the number of particles every process needs to analyse
particlesPerProcess = math.ceil(noOfParticles / noOfProcs)
#create noOfProcs Processes, each with a different set of particles
p = Process(target=processBatch, args=(
particles[i*particlesPerProcess:(i+1)*particlesPerProcess],
centerCoords, #handle to shared memory
totalMass, #handle to shared memory
lock, #handle to lock
'batch'+str(i)), #also pass name of process for easier logging
name='batch'+str(i))
processes.append(p)
print('created proc:',i)
#start all processes
for p in processes:
p.start() #here, the program waits for the started process to terminate. why?
#wait for all processes to finish
for p in processes:
p.join()
#normalize the coordinates
centerCoords[0] /= totalMass.value
centerCoords[1] /= totalMass.value
centerCoords[2] /= totalMass.value
print(centerCoords[:])
print('total time used', time() - startingTime, ' seconds')
class Particle():
"""a particle is a very simple physical object, having a set of x/y/z coordinates and a mass.
All values are randomly set at initialization of the object"""
def __init__(self):
self.x = random() * 1000
self.y = random() * 1000
self.z = random() * 1000
self.m = random() * 10
def printProperties(self):
attrs = vars(self)
print ('\n'.join("%s: %s" % item for item in attrs.items()))
def processBatch(particles,centerCoords,totalMass,lock,name):
"""calculates the mass-weighted sum of all coordinates of all particles as well as the sum of all masses.
Writes the results into the shared memory centerCoords and totalMass, using lock"""
print(name,' started')
mass = 0
centerX = 0
centerY = 0
centerZ = 0
for p in particles:
centerX += p.m*p.x
centerY += p.m*p.y
centerZ += p.m*p.z
mass += p.m
with lock:
centerCoords[0] += centerX
centerCoords[1] += centerY
centerCoords[2] += centerZ
totalMass.value += mass
print(name,' ended')
if __name__ == '__main__':
exercise2(2**16,6)
现在我希望所有进程大约在同一时间启动并并行执行。但是当我查看程序的输出时,这看起来好像进程正在串行执行:
created proc: 0
created proc: 1
created proc: 2
created proc: 3
created proc: 4
created proc: 5
batch0 started
batch0 ended
batch1 started
batch1 ended
batch2 started
batch2 ended
batch3 started
batch3 ended
batch4 started
batch4 ended
batch5 started
batch5 ended
[499.72234074100135, 497.26586187539453, 498.9208784328791]
total time used 4.7220001220703125 seconds
此外,当使用 Eclipse 调试器单步执行程序时,我可以看到程序总是等待一个进程终止,然后在标有以“为什么?”结尾的注释的行开始下一个进程。当然,这可能只是调试器,但是当我查看正常运行中产生的输出时,这正是上图所示。
- 这些进程是否并行执行而我只是因为标准输出的一些共享问题而看不到它?
- 如果进程是串行执行的:为什么?我怎样才能让它们并行运行?
非常感谢您对理解这一点的任何帮助。
我在具有双核 Intel 处理器的 Windows 7 机器上使用 Python 3.2.3 从 PyDev 和命令行执行了上述代码。
编辑:
由于程序的输出,我误解了这个问题:进程实际上是并行运行的,但是腌制大量数据并将其发送到子进程的开销需要很长时间,以至于完全扭曲了画面。
将粒子(即数据)的创建移至子进程,以便它们不必首先被腌制,从而消除了所有问题,并导致程序的有用的并行执行。
因此,为了解决这个任务,我必须将粒子保存在共享内存中,这样它们就不必传递给子进程。