[Python-Dev] Fwd: subprocess.Popen(.... stdout=IGNORE, ...)

[Josiah Carlson]

There is a related bit of functionality for subprocess that would allow
for asynchronous handling of IO to/from the called subprocess.  It has
been implemented as a recipe [1], but requires the use of additional
pywin32 functionality on Windows.  As was the case for the original
subprocess module, in order to get the proper functionality on Windows,
we may need to include additional features from pywin32 into the
_subprocess.c driver, or alternatively, convert all _subprocess.c bits
into ctypes calls.

If the inclusion of additional code into _subprocess.c or the use of
ctypes is undesireable, this feature could require the *user* to install
pywin32 on Windows, which would be unfortunate, but perfectly reasonable.

With an asynchronous handler for the subprocess module, a user could
ignore or process output from a subprocess as desired or necessary.

 - Josiah

[1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Peter Astrand <astrand at lysator.liu.se> wrote:
> 
> On Tue, 13 Jun 2006, Martin Blais wrote:
> 
> Hi all. Now let's see if I remember something about my module...
> 
> 
> > In the subprocess module, by default the files handles in the child
> > are inherited from the parent.  To ignore a child's output, I can use
> > the stdout or stderr options to send the output to a pipe::
> >
> >    p = Popen(command, stdout=PIPE, stderr=PIPE)
> >
> > However, this is sensitive to the buffer deadlock problem, where for
> > example the buffer for stderr might become full and a deadlock occurs
> > because the child is blocked on writing to stderr and the parent is
> > blocked on reading from stdout or waiting for the child to finish.
> >
> > For example, using this command will cause deadlock::
> >
> >    call('cat /boot/vmlinuz'.split(), stdout=PIPE, stderr=PIPE)
> 
> Yes, the call convenience function is basically for the case when you are
> not interested in redirection.
> 
> 
> > Popen.communicate() implements a solution using either select() or
> > multiple threads (under Windows) to read from the pipes, and returns
> > the strings as a result.  It works out like this::
> >
> >    p = Popen(command, stdout=PIPE, stderr=PIPE)
> >    output, errors = p.communicate()
> >    if p.returncode != 0:
> >         ?
> >
> > Now, as a user of the subprocess module, sometimes I just want to
> > call some child process and simply ignore its output, and to do so I
> > am forced to use communicate() as above and wastefully capture and
> > ignore the strings.  This is actually quite a common use case.  "Just
> > run something, and check the return code".
> 
> Yes, this is a common case, and using communicate() is indeed overkill and
> wasteful.
> 
> 
> > Right now, in order to do
> > this without polluting the parent's output, you cannot use the call()
> > convenience (or is there another way?).
> >
> > A workaround that works under UNIX is to do this::
> >
> >    FNULL = open('/dev/null', 'w')
> >    returncode = call(command, stdout=FNULL, stderr=FNULL)
> 
> Yes, this works. You can also do:
> 
> returncode = subprocess.call(command, stdout=open('/dev/null', 'w'), stderr=subprocess.STDOUT)
> 
> 
> > Some feedback requested, I'd like to know what you think:
> >
> > 1. Would it not be nice to add a IGNORE constant to subprocess.py
> >    that would do this automatically?, i.e. ::
> >
> >      returncode = call(command, stdout=IGNORE, stderr=IGNORE)
> >
> >    Rather than capture and accumulate the output, it would find an
> >    appropriate OS-specific way to ignore the output (the /dev/null file
> >    above works well under UNIX, how would you do this under Windows?
> >    I'm sure we can find something.)
> 
> I have a vague feeling of that this has been discussed before, but I
> cannot find a tracker for this. I guess an IGNORE constant would be nice.
> Using open('/dev/null', 'w') is only a few more characters to type, but as
> you say, it's not platform independent.
> 
> So, feel free to submit a patch or a Feature Request Tracker.
> 
> 
> > 2. call() should be modified to not be sensitive to the deadlock
> >    problem, since its interface provides no way to return the
> >    contents of the output.  The IGNORE value provides a possible
> >    solution for this.
> 
> How do you suggest the call() should be modified? I'm not really sure it
> can do more without being more complicated. Being simple is the main
> purpose of call().
> 
> 
> > 3. With the /dev/null file solution, the following code actually
> >    works without deadlock, because stderr is never blocked on writing
> >    to /dev/null::
> >
> >      p = Popen(command, stdout=PIPE, stderr=IGNORE)
> >      text = p.stdout.read()
> >      retcode = p.wait()
> >
> >    Any idea how this idiom could be supported using a more portable
> >    solution (i.e. how would I make this idiom under Windows, is there
> >    some equivalent to /dev/null)?
> 
> Yes, as Terry Reedy points out, NUL: can be used.
> 
> Regards,
> /Peter Åstrand <astrand at lysator.liu.se>
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jcarlson%40uci.edu