Do subprocess.PIPE and subprocess.STDOUT sametime

[Thomas Passin]

On 5/9/2023 2:13 PM, Horst Koiner wrote:
> Hi @all,
> i'm running a program which is still in development with subprocess.run (Python version 3.10), further i need to capture the output of the program in a python variable. The program itself runs about 2 minutes, but it can also freeze in case of new bugs.
> 
> For production i run the program with stdout=subprocess.PIPE and i can fetch than the output later. For just testing if the program works, i run with stdout=subprocess.STDOUT and I see all program output on the console, but my program afterwards crashes since there is nothing captured in the python variable. So I think I need to have the functionality of subprocess.PIPE and subprcess.STDOUT sametime.
> 
> What I tried until now:
> 1. Poll the the output and use Popen instead:
> 
> # Start the subprocess
> process = subprocess.Popen(['./test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
> 
> captured_output = b''
> process_running = True
> while process_running:
>      process_running = (process.poll() is not None)
>      for pipe in [ process.stdout, process.stderr ]:
>          while line := pipe.readline():
>              print(line)
>              captured_output += line
> 
> print(captured_output)
> return_code = process.returncode
> 
> => But this is discouraged by the python doc, since it says that polling this way is prone to deadlocks. Instead it proposes the use of the communicate() function.
> 
> 2. Use communicate() with timeout.
> => This works not at all since when the timeout occurs an exception is thrown and communicate returns at all.
> 
> 3. Use threading instead
> => For being that simple and universal like subprocess you will more or less reimplement subprocess with threading, like its done in subprocess.py. Just for a debug output the effort is much to high.
> 
> #######################################################
> Do you have further ideas for implementing such a behavior?
> Do you think that a feature request should be done of I'm omitting something obvious?

I'm not sure if this exactly fits your situation, but if you use 
subprocess with pipes, you can often get a deadlock because the stdout 
(or stderr, I suppose) pipe has a small capacity and fills up quickly 
(at least on Windows), then it blocks until it is emptied by a read. 
But if you aren't polling, you don't know there is something to read so 
the pipe never gets emptied.  And if you don't read it before the pipe 
has filled up, you may lose data.

I solved that by running communicate() on a separate thread.  Let the 
communicate block the thread until the process has completed, then have 
the thread send the result back to the main program.  Of course, this 
won't work if your process doesn't end since you won't get results until 
the process ends.