multiprocessing, queue

[Chris Angelico]

On Sat, May 9, 2015 at 12:31 AM, Michael Welle <mwe012008 at gmx.net> wrote:
>> As a general rule, queues need to have both ends operating
>> simultaneously, otherwise you're likely to have them blocking. In
>> theory, your code should all work with ridiculously low queue sizes;
>> the only cost will be concurrency (since you'd forever be waiting for
>> the queue, so your tasks will all be taking turns). I tested this by
>> changing the Queue() calls to Queue(1), and the code took about twice
>> as long to complete. :)
> ;) I know, as you might guess it's not a real world example. It's just
> to explore the multiprocessing module.

Sure, but even in a real-world example, it shouldn't ever be necessary
to create huge queues. Larger queues allow for inconsistent
performance of producer and/or consumer (eg if your consumer takes 1s
to do each of 500 jobs, then 500s to do one job, it's capable of
coping with a producer that puts one job on the queue every 2s, but
only if the queue can handle ~250 jobs), but otherwise, the only
effect of shrinking the queues is to force the processes into
lock-step. Bigger queues mean that an over-performing producer will
run you out of memory rather than get blocked. At very least, it
should be a safe way to debug your logic - cut the queues down to just
2-3 elements each, and keep on printing out what's in the queue.

ChrisA