Getting stdout and stderr from subprocess in correct order

[Ivan "Rambius" Ivanov]

Hello,

Thank you all for your suggestions. I will see what will apply to my use case.

Regards
Rambius

On Sat, Mar 4, 2017 at 5:37 PM, Piet van Oostrum
<piet-l at pietvanoostrum.com> wrote:
> "Ivan \"Rambius\" Ivanov" <rambiusparkisanius at gmail.com> writes:
>
>> Dear colleagues,
>>
>> I using subprocess module and I am wondering how I can get the output
>> of the spawned process's stdout and stderr in the right order. Here
>> are my sample programs:
>>
>> $ cat subprc.py
>> import subprocess
>> import sys
>>
>> f = 'hw.py'
>> p = subprocess.run([sys.executable, f], stdout=subprocess.PIPE,
>> stderr=subprocess.PIPE)
>> print(p.stdout)
>> print(p.stderr)
>>
>> $ cat hw.py
>> import sys
>>
>> print("a", file=sys.stdout)
>> print("b", file=sys.stderr)
>> print("c", file=sys.stdout)
>> print("d", file=sys.stderr)
>>
>> When i run hw.py I get
>>
>> $ ./hw.py
>> a
>> b
>> c
>> d
>>
>> When I run it through subprc.py, I do get standard out and standard
>> errors streams, but they are separated and not in the order above:
>>
>> $ ./subprc.py
>> b'a\nc\n'
>> b'b\nd\n'
>>
>> How should I use subprocess in order to get the outputs in the correct
>> order? Thank you for your help in advance.
>>
> If you want them in the order they are produced then you have to put
> them through a single channel with stderr=subprocess.STDOUT.
>
> p = subprocess.run([sys.executable, f], stdout=subprocess.PIPE,
> stderr=subprocess.STDOUT, encoding='ascii')
>
> I added an encoding so that it will be read as a text file rather than
> binary. And of course you should read only stdout (stderr will give
> None)
>
> Moreover you must make sure that each write is flushed. This can be done
> in three ways:
>
> 1. Add the flush=True argument to the print statements.
> print("a", file=sys.stdout, flush=True)
> print("b", file=sys.stderr, flush=True)
> print("c", file=sys.stdout, flush=True)
> print("d", file=sys.stderr, flush=True)
>
> 2. Change the files to a line buffered version.
> import os
> sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)
> sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 1)
>
> 3. Add a flush() call after each print statement, like
> sys.stdout.flush()
>
> Then you get them in order.
> $ python3 subprc.py
> a
> b
> c
> d
>
> Of course you won't be able to tell from which stream each line comes.
> The streams are byte streams, not message streams. You would have to put
> extra information, e.g. a prefix, in your print statements.
> --
> Piet van Oostrum <piet-l at pietvanoostrum.com>
> WWW: http://pietvanoostrum.com/
> PGP key: [8DAE142BE17999C4]
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Tangra Mega Rock: http://www.radiotangra.com