70% [* SPAM *] multiprocessing.Queue blocks when sending large object

DPalao dpalao.python at gmail.com
Tue Nov 29 14:09:44 EST 2011


Hello,
I'm trying to use multiprocessing to parallelize a code. There is a number of 
tasks (usually 12) that can be run independently. Each task produces a numpy 
array, and at the end, those arrays must be combined.
I implemented this using Queues (multiprocessing.Queue): one for input and 
another for output.
But the code blocks. And it must be related to the size of the item I put on 
the Queue: if I put a small array, the code works well; if the array is 
realistically large (in my case if can vary from 160kB to 1MB), the code 
blocks apparently forever.
I have tried this:
http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-large-
objects/
but it didn't work (especifically I put a None sentinel at the end for each 
worker).

Before I change the implementation, 
is there a way to bypass this problem with  multiprocessing.Queue?
Should I post the code (or a sketchy version of it)?

TIA,

David



More information about the Python-list mailing list