multiprocessing queue hangs up on the Amazon cloud

[Chris Kaynor]

On Wed, Jan 14, 2015 at 2:16 PM, Chris Angelico <rosuav at gmail.com> wrote:

> And then you seek to run multiple workers. If my reading is correct,
> one of them (whichever one happens to get there first) will read the
> STOP marker and finish; the others will all be blocked, waiting for
> more work (which will never arrive). In theory, you could have the
> first process be the one to stop, in which case you'd successfully
> join() it and then go on to wait for the second, but if any other
> process gets it, you'll be stuck waiting for the first. Or have I
> misunderstood something in your logic?
>
> If this is indeed what's happening, the simplest solution might be to
> add as many STOPs as you have workers. Alternatively, if you can
> guarantee that the work is all on the queue before the first process
> starts, you could simply use the empty queue as the sentinel; which I
> would recommend doing for the done_queue, as there's only one reader
> for that.
>
> But it's entirely possible I've missed some tiny fact that makes my
> entire analysis wrong, in which case I apologize for the noise!
>

My reading of the code is the same as Chris Angelico's, however I would
like to add some more:

Another solution to the STOP issue would be to have each worker re-add the
STOP to the work queue once it is complete. This is a good solution if you
cannot put all the work on the queues before starting the workers or the
number of workers is not known when creating the work queue. This can be
done by merely adding the line "work_queue.put('STOP')" to the end of the
worker function. Note that this solution will not work correctly if workers
may produce more work (but, then again, neither will the STOP entry).

I'd also like to point out a flaw in the results processing, at the end of
the worker process you have "done_queue.put('STOP')", however one thread
may run that before another has added its final work result to the
done_queue. That will cause the following two lines:
On Thu, Jan 15, 2015 at 8:55 AM,  <jgrant at smith.edu> wrote:

>         for status in iter(done_queue.get, 'STOP'):
>                 print status

to exit early, and not print the status of some of the work items.

For this, Chris A's other option is the best: due to the join loop just
above, you know all items have been entered into the done_queue, and thus,
there is no need for the STOP entry: just run until the queue has been
fully processed.

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150114/f7b677c5/attachment.html>