Python deadlock using subprocess.popen and communicate

[Nobody]

On Thu, 22 Sep 2011 08:55:40 -0700, Atherun wrote:

>> Just handle process.stdout/stderr by yourself - read it out until EOF
>> and then wait() for the process.
> 
> Thats what confuses me though, the documentation says
> process.stdout.read()/stderr.read() can deadlock and apparently so can
> communicate, how do you read the stdout/stderr on yourself if its
> documented using them can cause a deadlock?

If you try to read/write two or more of stdin/stdout/stderr via the
"naive" approach, you run the risk of the child process writing more than
a pipe's worth of data to one stream (and thus blocking) while the
parent is performing a blocking read/write on another stream, resulting in
deadlock.

The .communicate() method avoids the deadlock by either:

1. On Unix, using non-blocking I/O and select(), or
2. On Windows, creating a separate thread for each stream.

Either way, the result is that it can always read/write whichever
streams are ready, so the child will never block indefinitely while
waiting for the parent.

If .communicate() is blocking indefinitely, it suggests that the child
process never terminates. There are many reasons why this might happen,
and most of them depend upon exactly what the child process is doing.

I suggest obtaining a copy of Process Explorer, and using it to
investigate the state of both processes (but especially the child) at the
point that the "deadlock" seems to occur.