Do subprocess.PIPE and subprocess.STDOUT sametime

[Mats Wichmann]

On 5/9/23 12:13, Horst Koiner wrote:
> Hi @all,
> i'm running a program which is still in development with subprocess.run (Python version 3.10), further i need to capture the output of the program in a python variable. The program itself runs about 2 minutes, but it can also freeze in case of new bugs.
> 
> For production i run the program with stdout=subprocess.PIPE and i can fetch than the output later. For just testing if the program works, i run with stdout=subprocess.STDOUT and I see all program output on the console, but my program afterwards crashes since there is nothing captured in the python variable. So I think I need to have the functionality of subprocess.PIPE and subprcess.STDOUT sametime.

I'm not sure you quite understood what subprocess.STDOUT is for.  If you 
say nothing stdout is not captured. STDOUT is used as a value for stderr 
to mean send it the same place as stdout, which is useful if you set 
stdout to something unusual, then you don't have to retype it if you 
want stderr going the same place.  The subprocess module, afaik, doesn't 
even have a case for stdout=STDOUT.

> 
> What I tried until now:
> 1. Poll the the output and use Popen instead:
> 
> # Start the subprocess
> process = subprocess.Popen(['./test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
> 
> captured_output = b''
> process_running = True
> while process_running:
>      process_running = (process.poll() is not None)
>      for pipe in [ process.stdout, process.stderr ]:
>          while line := pipe.readline():
>              print(line)
>              captured_output += line
> 
> print(captured_output)
> return_code = process.returncode
> 
> => But this is discouraged by the python doc, since it says that polling this way is prone to deadlocks. Instead it proposes the use of the communicate() function.
> 
> 2. Use communicate() with timeout.
> => This works not at all since when the timeout occurs an exception is thrown and communicate returns at all.

Well, sure ... if you set timeout, then you need to be prepared to catch 
the TimeoutExpired exception and deal with it. That should be entirely 
normal.

> 
> 3. Use threading instead
> => For being that simple and universal like subprocess you will more or less reimplement subprocess with threading, like its done in subprocess.py. Just for a debug output the effort is much to high.

Not sure I get what this is asking/suggesting.  If you don't want to 
wait for the subprocess to run, you can use async - that's been fully 
implemented.

https://docs.python.org/3/library/asyncio-subprocess.html