Dealing with errors in interactive subprocess running python interpreter that freeze the process

[Chris Angelico]

On Thu, Aug 2, 2018 at 6:46 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On Wed, 1 Aug 2018 at 21:17, <cseberino at gmail.com> wrote:
>>
>> I can run python3 interactively in a subprocess w/ Popen but
>> if I sent it text, that throws an exception, the process freezes
>> instead of just printing the exception like the normal interpreter..
>> why? how fix?  Here is my code below.
>>
>> (I suspect when there is an exception, there is NO output to stdin so that
>> the problem is the line below that tries to read from stdin never finishes.
>> Maybe I need a different readline that can "survive" when there is no output and won't block?)
>>
>> ....
>>
>> import subprocess
>>
>> interpreter = subprocess.Popen(['python3', '-i'],
>>                                stdin  = subprocess.PIPE,
>>                                stdout = subprocess.PIPE,
>>                                stderr = subprocess.PIPE)
>>
>> while True:
>>         exp = input(">>> ").encode() + b"\n"
>>         interpreter.stdin.write(exp)
>>         interpreter.stdin.flush()
>>         print(interpreter.stdout.readline().strip())
>> interpreter.stdin.close()
>> interpreter.terminate()
>
> You're only reading one line from stdout, but an exception is multiple
> lines. So the subprocess is still trying to write while you're wanting
> to give it input again. This is a classic way to get a deadlock, which
> is basically what you're seeing. Add to that the fact that there are
> likely IO buffers in the subprocess that mean it's not necessarily
> passing output back to you at the exact time you expect it to (and the
> subprocess probably has different buffering behaviour when the IO is
> to pipes rather than to the console) and it gets complex fast.
>
> As others have mentioned, separate threads for the individual pipes
> may help, or if you need to go that far there are specialised
> libraries, I believe (pexpect is one, but from what I know it's fairly
> Unix-specific, so I'm not very familiar with it).

Another possibility: If the ONLY thing you're doing with stdout/stderr
is passing them through to the screen, simply don't change them. Let
them remain bound to the console. You can have a pipe for stdin
without also having pipes for the others. But that won't work if you
intend to do any sort of parsing on the returned output.

ChrisA