multiprocessing Pool workers cannot create subprocesses

[Jason Grout]

On 3/19/11 4:17 PM, John L. Stephens wrote:
> On 3/18/2011 7:54 PM, Jason Grout wrote:
>>
>> Right; thanks. Let me rephrase my questions:
>>
>> 1. Why is important that the multiprocessing Pool worker processors
>> have daemon=True (I think this is the same as asking: why is it
>> important that they be terminated with terminate() rather than join() )?
>>
>> 2. Is it safe for us to reset a Pool worker process to have
>> daemon=False before starting a subprocess from that worker, like in
>> the code from the original message?
>>
>> Thanks,
>>
>> Jason
> Jason,
>
> I just happen to be dealing with a project that uses multiprocessing.
>
> What I have learned is this...
>
> If a child thread (pool worker) is not set to daemon (daemon=False), if
> for some reason the parent thread terminates either normally or
> abnormally and the worker thread has not completed its task, the child
> thread will terminate by throwing all sorts of nasty errors. However, in
> daemon mode, multiprocessing will terminate the child thread 'cleanly'.
> That's not to say that the worker has a chance to complete its work or
> shut itself down. Multiprocessing will absorb the exceptions and not
> pass them along.
>
> You may terminate a child thread using join(). That is probably the
> safest way to do it. terminate() will just kill the worker thread
> immediately without any regard to whether or not it has completed its
> tasks. I believe multiprocessing uses terminate() as well to kill a
> daemon thread if the parent thread disappears.
>
> join() will, however, block until the task has competed and returned. If
> you want to continue doing work in the parent thread while the child
> thread is off doing its thing, then another means of syncing up the
> parent and children threads is needed.
>
> As for allowing children threads to spawn off children of its own using
> subprocess runs the risk of creating a little army of zombie
> 'grandchildren' if either the parent or child threads terminate before
> the subprocess completes and returns.
>
> Hope that helps....

Thanks.  That helps tremendously!

Jason