Getting stdout and stderr from subprocess in correct order

[dieter]

"Ivan \"Rambius\" Ivanov" <rambiusparkisanius at gmail.com> writes:
> ...
> I using subprocess module and I am wondering how I can get the output
> of the spawned process's stdout and stderr in the right order. Here
> are my sample programs:
>
> $ cat subprc.py
> import subprocess
> import sys
>
> f = 'hw.py'
> p = subprocess.run([sys.executable, f], stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> print(p.stdout)
> print(p.stderr)
> ...
> How should I use subprocess in order to get the outputs in the correct
> order? Thank you for your help in advance.

If the order really is important and it is not important
from which stream the data comes, then you can try
to map "stdout" and "stderr" to the same file.
Note that this still may not give you the hoped for result:
"stderr" is typically written unbuffered, "stdout" is typically written
either "line buffered" (output to a terminal) or "fully buffered".
Thus, you can see "stdout" output only after the buffer content is
given over to the operationg system (when the buffer is full, explicitely
flushed or the file closed).

To do this with "subprocess", you need to create a pipe
outside the "subprocess.run" (--> "r_fd, w_fd = os.pipe()")
and pass the output part ("w_fd") to the "run".
You can then read the input part ("r_fd"). Note, that
the "*_fd" are file descriptors (represented as integers), not
Python "file" objects. You can use either "os.read" (to read the
input directly) or "os.fdopen" (to turn a file descriptor into a Python
file object).

Alternatively, you can spawn two threads for reading input from
"stdout" and "stderr", respectively - as soon as one becomes
available and merge them in a data structure of your own.
Then, you can tag the data with an indication from which stream
the output came.
Note, the remarks about buffering in the first paragraphe of my response
(you can read only what has been previously given over to the operating
system -- things still in an output buffer cannot be accessed).