Dealing with errors in interactive subprocess running python interpreter that freeze the process

[Paul Moore]

On Wed, 1 Aug 2018 at 21:17, <cseberino at gmail.com> wrote:
>
> I can run python3 interactively in a subprocess w/ Popen but
> if I sent it text, that throws an exception, the process freezes
> instead of just printing the exception like the normal interpreter..
> why? how fix?  Here is my code below.
>
> (I suspect when there is an exception, there is NO output to stdin so that
> the problem is the line below that tries to read from stdin never finishes.
> Maybe I need a different readline that can "survive" when there is no output and won't block?)
>
> ....
>
> import subprocess
>
> interpreter = subprocess.Popen(['python3', '-i'],
>                                stdin  = subprocess.PIPE,
>                                stdout = subprocess.PIPE,
>                                stderr = subprocess.PIPE)
>
> while True:
>         exp = input(">>> ").encode() + b"\n"
>         interpreter.stdin.write(exp)
>         interpreter.stdin.flush()
>         print(interpreter.stdout.readline().strip())
> interpreter.stdin.close()
> interpreter.terminate()

You're only reading one line from stdout, but an exception is multiple
lines. So the subprocess is still trying to write while you're wanting
to give it input again. This is a classic way to get a deadlock, which
is basically what you're seeing. Add to that the fact that there are
likely IO buffers in the subprocess that mean it's not necessarily
passing output back to you at the exact time you expect it to (and the
subprocess probably has different buffering behaviour when the IO is
to pipes rather than to the console) and it gets complex fast.

As others have mentioned, separate threads for the individual pipes
may help, or if you need to go that far there are specialised
libraries, I believe (pexpect is one, but from what I know it's fairly
Unix-specific, so I'm not very familiar with it).

Sorry, but there's no "simple" answer here for you (although you may
well be able to get something that works well enough for your specific
needs - but it's not obvious from your snippet of code what you're
trying to achieve).

Paul