[Python-Dev] A question about the subprocess implementation

[Terry Reedy]

On 1/7/2012 4:25 PM, Vinay Sajip wrote:
> The subprocess.Popen constructor takes stdin, stdout and stderr keyword
> arguments which are supposed to represent the file handles of the child process.
> The object also has stdin, stdout and stderr attributes, which one would naively
> expect to correspond to the passed in values, except where you pass in e.g.
> subprocess.PIPE (in which case the corresponding attribute would be set to an
> actual stream or descriptor).
>
> However, in common cases, even when keyword arguments are passed in, the
> corresponding attributes are set to None. The following script
>
> import os
> from subprocess import Popen, PIPE
> import tempfile
>
> cmd = 'ls /tmp'.split()
>
> p = Popen(cmd, stdout=open(os.devnull, 'w+b'))
> print('process output streams: %s, %s' % (p.stdout, p.stderr))
> p = Popen(cmd, stdout=tempfile.TemporaryFile())
> print('process output streams: %s, %s' % (p.stdout, p.stderr))
>
> prints
>
> process output streams: None, None
> process output streams: None, None
>
> under both Python 2.7 and 3.2. However, if subprocess.PIPE is passed in, then
> the corresponding attribute *is* set: if the last four lines are changed to
>
> p = Popen(cmd, stdout=PIPE)
> print('process output streams: %s, %s' % (p.stdout, p.stderr))
> p = Popen(cmd, stdout=open(os.devnull, 'w+b'), stderr=PIPE)
> print('process output streams: %s, %s' % (p.stdout, p.stderr))
>
> then you get
>
> process output streams:<open file '<fdopen>', mode 'rb' at 0x2088660>, None
> process output streams: None,<open file '<fdopen>', mode 'rb' at 0x2088e40>
>
> under Python 2.7, and
>
> process output streams:<_io.FileIO name=3 mode='rb'>, None
> process output streams: None,<_io.FileIO name=5 mode='rb'>
>
> This seems to me to contradict the principle of least surprise. One would
> expect, when an file-like object is passed in as a keyword argument, that it be
> placed in the corresponding attribute.

The behavior matches the doc: Popen.stdin
If the stdin argument was PIPE, this attribute is a file object that 
provides input to the child process. Otherwise, it is None.
-- ditto for Popen.stdout, .stderr

> That way, if one wants to do
> p.stdout.close() (which is necessary in some cases), one doesn't hit an
> AttributeError because NoneType has no attribute 'close'.

I believe you are expected to keep a reference to anything you pass in.

pout = open(os.devnull, 'w+b')
p = Popen(cmd, stdout=pout, 'w+b'), stderr=PIPE)

The attributes were added for the case when you do not otherwise have 
access.

> This seems like it might be a bug, but if so it does seem rather egregious:

It would be egregious if is were a bug, but it is not.

> someone tell me if there is a good design reason for the current behaviour? If
> there isn't one, I'll raise an issue.

That seems like a possibly reasonable enhancement request. But the 
counterargument might be that you have to separately keep track of the 
need to close anyway. Or that you should do things like

with open(os.devnull, 'w+b') as pout:
     p = Popen(cmd, stdout=pout, 'w+b'), stderr=PIPE)

-- 
Terry Jan Reedy