Do subprocess.PIPE and subprocess.STDOUT sametime

[Horst Koiner]

Hi @all,
i'm running a program which is still in development with subprocess.run (Python version 3.10), further i need to capture the output of the program in a python variable. The program itself runs about 2 minutes, but it can also freeze in case of new bugs.

For production i run the program with stdout=subprocess.PIPE and i can fetch than the output later. For just testing if the program works, i run with stdout=subprocess.STDOUT and I see all program output on the console, but my program afterwards crashes since there is nothing captured in the python variable. So I think I need to have the functionality of subprocess.PIPE and subprcess.STDOUT sametime.

What I tried until now:
1. Poll the the output and use Popen instead:

# Start the subprocess
process = subprocess.Popen(['./test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

captured_output = b''
process_running = True
while process_running:
    process_running = (process.poll() is not None)
    for pipe in [ process.stdout, process.stderr ]:
        while line := pipe.readline():
            print(line)
            captured_output += line

print(captured_output)
return_code = process.returncode 

=> But this is discouraged by the python doc, since it says that polling this way is prone to deadlocks. Instead it proposes the use of the communicate() function.

2. Use communicate() with timeout.
=> This works not at all since when the timeout occurs an exception is thrown and communicate returns at all.

3. Use threading instead
=> For being that simple and universal like subprocess you will more or less reimplement subprocess with threading, like its done in subprocess.py. Just for a debug output the effort is much to high.

#######################################################
Do you have further ideas for implementing such a behavior?
Do you think that a feature request should be done of I'm omitting something obvious?

Thanks you in advance for your suggestions,
Horst.