Do subprocess.PIPE and subprocess.STDOUT sametime

[Chris Angelico]

On Sat, 13 May 2023 at 07:21, Horst Koiner <koinerhorst6 at gmail.com> wrote:
>
> Using asyncio for this is a good possibility I was not aware of.
>
> My best try with asyncio was:
> import asyncio
>
> async def run_command():
>     # Create subprocess
>     process = await asyncio.create_subprocess_exec(
>         './test.sh',
>         stdout=asyncio.subprocess.PIPE,  # Redirect stdout to a pipe
>         stderr=asyncio.subprocess.PIPE  # Redirect stderr to a pipe
>     )
>
>     # Read stdout and stderr asynchronously
>     captured_output = b''
>     async for line in process.stdout:
>         print(line.decode().strip())
>         captured_output += line
>     async for line in process.stderr:
>         print(line.decode().strip())
>         captured_output += line
>
>     await process.wait()
>     print(captured_output)
>
>
> # Run the asyncio event loop
> asyncio.run(run_command())
> ########################################
>
> This fulfills all my requirements. A nice to have would that the captured_output has not to be constructed with += 's but with a final seek(0) and read() of process.stdout. But I didn't find anything how to rewind the stream, that i can read the whole output again.
> Another question is, if this solution is deadlock proof.
>

No it's not, but the best part is, it's really close to! Asynchronous
I/O is perfect for this: you need to wait for any of three events
(data on stdout, data on stderr, or process termination). So here it
is as three tasks:

captured_output = b""
async def collect_output(stream):
    global captured_output
    async for line in stream:
        print(line.decode().strip())
        captured_output += line

(You can play around with other ways of scoping this, I'm just using a
global for simplicity)

Inside run_command, you can then spawn three independent tasks and
await them simultaneously. Once all three finish, you have your
captured output, and the process will have terminated.

This would then be guaranteed deadlock-proof (and, if you needed to
feed data on stdin, you could do that with a fourth task and it'd
still be deadlock-proof, even if it's more than one pipe buffer of
input), since all the pipes are being managed concurrently.

Even cooler? You can scale this up to multiple processes by calling
run_command more than once as separate tasks!

ChrisA