Getting stdout and stderr from subprocess in correct order

[Piet van Oostrum]

"Ivan \"Rambius\" Ivanov" <rambiusparkisanius at gmail.com> writes:

> Dear colleagues,
>
> I using subprocess module and I am wondering how I can get the output
> of the spawned process's stdout and stderr in the right order. Here
> are my sample programs:
>
> $ cat subprc.py
> import subprocess
> import sys
>
> f = 'hw.py'
> p = subprocess.run([sys.executable, f], stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> print(p.stdout)
> print(p.stderr)
>
> $ cat hw.py
> import sys
>
> print("a", file=sys.stdout)
> print("b", file=sys.stderr)
> print("c", file=sys.stdout)
> print("d", file=sys.stderr)
>
> When i run hw.py I get
>
> $ ./hw.py
> a
> b
> c
> d
>
> When I run it through subprc.py, I do get standard out and standard
> errors streams, but they are separated and not in the order above:
>
> $ ./subprc.py
> b'a\nc\n'
> b'b\nd\n'
>
> How should I use subprocess in order to get the outputs in the correct
> order? Thank you for your help in advance.
>
If you want them in the order they are produced then you have to put
them through a single channel with stderr=subprocess.STDOUT.

p = subprocess.run([sys.executable, f], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, encoding='ascii')

I added an encoding so that it will be read as a text file rather than
binary. And of course you should read only stdout (stderr will give
None)

Moreover you must make sure that each write is flushed. This can be done
in three ways:

1. Add the flush=True argument to the print statements.
print("a", file=sys.stdout, flush=True)
print("b", file=sys.stderr, flush=True)
print("c", file=sys.stdout, flush=True)
print("d", file=sys.stderr, flush=True)

2. Change the files to a line buffered version.
import os
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)
sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 1)

3. Add a flush() call after each print statement, like
sys.stdout.flush()

Then you get them in order.
$ python3 subprc.py
a
b
c
d

Of course you won't be able to tell from which stream each line comes.
The streams are byte streams, not message streams. You would have to put
extra information, e.g. a prefix, in your print statements.
-- 
Piet van Oostrum <piet-l at pietvanoostrum.com>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]