os.wait() losing child?

[Hrvoje Niksic]

Nick Craig-Wood <nick at craig-wood.com> writes:

>>  I think your polling way works; it seems there no other way around this 
>>  problem other than polling or extending Popen class.
>
> I think polling is probably the right way of doing it...

It requires the program to wake up every 0.1s to poll for freshly
exited subprocesses.  That doesn't consume excess CPU cycles, but it
does prevent the kernel from swapping it out when there is nothing to
do.  Sleeping in os.wait allows the operating system to know exactly
what the process is waiting for, and to move it out of the way until
those conditions are met.  (Pedants would also notice that polling
introduces on average 0.1/2 seconds delay between the subprocess dying
and the parent reaping it.)

In general, a program that waits for something should do so in a
single call to the OS.  OP's usage of os.wait was exactly correct.

Fortunately the problem can be worked around by hanging on to Popen
instances until they are reaped.  If all of them are kept referenced
when os.wait is called, they will never end up in the _active list
because the list is only populated in Popen.__del__.

> Internally subprocess uses os.waitpid(pid) just waiting for its own
> specific pids.  IMHO this is the right way of doing it other than
> os.wait() which waits for any pids.  os.wait() can reap children
> that you weren't expecting (say some library uses os.system())...

system calls waitpid immediately after the fork.  This can still be a
problem for applications that call wait in a dedicated thread, but the
program can always ignore the processes it doesn't know anything
about.