[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly

[Ask Solem]

Ask Solem <askh at opera.com> added the comment:

> Does the problem make sense/do you have any ideas for an alternate
> solution?

Well, I still haven't given up on the trackjobs patch. I changed it to use a single queue for both the acks and the result (see new patch attached:  multiprocessing-trunk at 82502-termination-trackjobs2.patch)

Been running it in production for a few days now, and it seems to work. But the tests still hangs from time to time, it seems they hang more frequent now than in the first patch (this may actually be a good thing:)

Would you like to try and identify the cause of this hang? Still haven't been able to.

I'm not sure about the overhead of using one queue per process either, but I'm usually running about 8 processes per CPU core for IO bound jobs (adding more processes after that usually doesn't affect performance in positive ways). There's also the overhead of the synchronization (ACK). Not sure if this is important performance-wise, but at least this makes it harder for me to reason about the problem.

----------
Added file: http://bugs.python.org/file18657/multiprocessing-trunk@82502-termination-trackjobs2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9205>
_______________________________________