I'm working on a fairly large project in Python that requires one of the compute-intensive background tasks to be offloaded to another core, so that the main service isn't slowed down. I've come across some apparently strange behaviour when using multiprocessing.Queue to communicate results from the worker process. Using the same queue for both a threading.Thread and a multiprocessing.Process for comparison purposes, the thread works just fine but the process fails to join after putting a large item in the queue. Observe:
import threading
import multiprocessing
class WorkerThread(threading.Thread):
def __init__(self, queue, size):
threading.Thread.__init__(self)
self.queue = queue
self.size = size
def run(self):
self.queue.put(range(size))
class WorkerProcess(multiprocessing.Process):
def __init__(self, queue, size):
multiprocessing.Process.__init__(self)
self.queue = queue
self.size = size
def run(self):
self.queue.put(range(size))
if __name__ == "__main__":
size = 100000
queue = multiprocessing.Queue()
worker_t = WorkerThread(queue, size)
worker_p = WorkerProcess(queue, size)
worker_t.start()
worker_t.join()
print 'thread results length:', len(queue.get())
worker_p.start()
worker_p.join()
print 'process results length:', len(queue.get())
I've seen that this works fine for size = 10000, but hangs at worker_p.join() for size = 100000. Is there some inherent size limit to what multiprocessing.Process instances can put in a multiprocessing.Queue? Or am I making some obvious, fundamental mistake here?
For reference, I am using Python 2.6.5 on Ubuntu 10.04.