Python deadlock using subprocess.popen and communicate

[Atherun]

On Sep 22, 10:44 am, Nobody <nob... at nowhere.com> wrote:
> On Thu, 22 Sep 2011 08:55:40 -0700, Atherun wrote:
> >> Just handle process.stdout/stderr by yourself - read it out until EOF
> >> and then wait() for the process.
>
> > Thats what confuses me though, the documentation says
> > process.stdout.read()/stderr.read() can deadlock and apparently so can
> > communicate, how do you read the stdout/stderr on yourself if its
> > documented using them can cause a deadlock?
>
> If you try to read/write two or more of stdin/stdout/stderr via the
> "naive" approach, you run the risk of the child process writing more than
> a pipe's worth of data to one stream (and thus blocking) while the
> parent is performing a blocking read/write on another stream, resulting in
> deadlock.
>
> The .communicate() method avoids the deadlock by either:
>
> 1. On Unix, using non-blocking I/O and select(), or
> 2. On Windows, creating a separate thread for each stream.
>
> Either way, the result is that it can always read/write whichever
> streams are ready, so the child will never block indefinitely while
> waiting for the parent.
>
> If .communicate() is blocking indefinitely, it suggests that the child
> process never terminates. There are many reasons why this might happen,
> and most of them depend upon exactly what the child process is doing.
>
> I suggest obtaining a copy of Process Explorer, and using it to
> investigate the state of both processes (but especially the child) at the
> point that the "deadlock" seems to occur.

In the one case I can easily reproduce, its in a p4.exe call that I'm
making both python and p4.exe have nearly the same stack for their
threads:
python:

ntoskrnl.exe!memset+0x64a
ntoskrnl.exe!KeWaitForMultipleObjects+0xd52
ntoskrnl.exe!KeWaitForMutexObject+0x19f
ntoskrnl.exe!__misaligned_access+0xba4
ntoskrnl.exe!__misaligned_access+0x1821
ntoskrnl.exe!KeWaitForMultipleObjects+0xf5d
ntoskrnl.exe!KeWaitForMutexObject+0x19f
ntoskrnl.exe!NtWaitForSingleObject+0xde
ntoskrnl.exe!KeSynchronizeExecution+0x3a43
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x4a8
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x429
ntdll.dll!RtlUniform+0x6e6
ntdll.dll!RtlCreateTagHeap+0xa7
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!ZwWaitForSingleObject+0x15
kernel32.dll!WaitForSingleObjectEx+0x43
kernel32.dll!WaitForSingleObject+0x12
python26.dll!_Py_svnversion+0xcf8


p4:

ntoskrnl.exe!memset+0x64a
ntoskrnl.exe!KeWaitForMultipleObjects+0xd52
ntoskrnl.exe!KeWaitForSingleObject+0x19f
ntoskrnl.exe!_misaligned_access+0xba4
ntoskrnl.exe!_misaligned_access+0x1821
ntoskrnl.exe!KeWaitForMultipleObjects+0xf5d
ntoskrnl.exe!KeWaitForSingleObject+0x19f
ntoskrnl.exe!NtCreateFile+0x4c9
ntoskrnl.exe!NtWriteFile+0x7e3
ntoskrnl.exe!KeSynchronizeExecution+0x3a43
ntdll.dll!ZwWriteFile+0xa
KERNELBASE.dll!WriteFile+0x7b
kernel32.dll!WriteFile+0x36
p4.exe+0x42d4b
p4.exe+0x42ed8


To me it looks like they're both waiting on each other.