[Python-Dev] subprocess crossplatformness and async communication

[Guido van Rossum]

On Mon, Jan 26, 2009 at 12:44 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> You can't launch a process and communicate with it without blocking at
> some point. The scenario where you can periodically check output and
> error pipes without blocking and send input is not supported.
>
> Scenario One: control console program. This implies reading from input
> and writing replies asyncronously
> - Popen.stdin, stdout, stderr - stdin.write(), stdout.read() or
> stderr.read() block script execution if any of the other OS pipe
> buffers fills up and blocks the child process. Advice is to use
> communicate(), but it waits for process to exit and hence doesn't
> allow to send the input based on reply from previous send.
>
> Scenario Two: waiting for process that terminates with extensive output blocks
> - Popen.wait() blocks if the child process generates enough output to
> a stdout or stderr pipe, advice is to use Popen.communicate()
> http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
> - Popen.communicate() - Waits for process to terminate, reads data
> from stdout and stderr until end-of-file. Cashes data in memory -
> should not be used for if the data size is large or unlimited.
>
> Solution - read(maxsize) and write(maxsize) functions that will return
> immediately.

Hi Anatoly -- thanks for clarifying your issues. I hope other
developers more familiar with subprocess.py will chime in and either
help you figure out a way to do this without changes to the subprocess
module, or, if that would be too painful, help you develop additional
APIs. You could start by proposing a set of changes to subprocess.py
and submit them as a patch to bugs.python.org; that is easier to deal
with than pointing to a recipe on the Activestate site.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)