subprocess module and blocking

[Dieter Maurer]

Robin Becker <robin at SPAMREMOVEjessikat.fsnet.co.uk> writes on Sun, 12 Jun 2005 09:22:52 +0000:
> I'm using a polling loop in a thread that looks approximately like this
> 
> while 1:
>      p = find_a_process()
>      rc = p.poll()
>      if rc is not None:
>          out, err = p.communicate()
>          #deal with output etc
>      sleep(1)
>....
> I notice that under both win32 and freebsd that things are fine
> provided that the subprocess doesn't write too much to
> stdout/stderr. However, the subprocess seems to lock often if too much
> is written (under freebsd I see a process state of POLL). I assume
> that the subprocess is filling up the pipe and then failing to wake up
> again. I had expected that subprocess would take care of this for me,

You just found out that this is not the case.

The warning attached to "communicate"s docstring might have
you averted: "Note: the data read is buffered in memory...
do not use for large size".

If subprocess would do what you expect, it would need to
buffer the output (to make room in the pipes again).
But, for large data, this could have dramatic consequences.

Thus, you should use "select" on the pipes to find out
when to read data.


Dieter