Python3 "pickle" vs. stdin/stdout - unable to get clean byte streams in Python 3

[Peter Otten]

John Nagle wrote:

>   I have working code from Python 2 which uses "pickle"
> to talk to a subprocess via stdin/stdio.  I'm trying to
> make that work in Python 3.
> 
>   First, the subprocess Python is invoked with the "-d' option,
> so stdin and stdio are supposed to be unbuffered binary streams.
> That was enough in Python 2, but it's not enough in Python 3.
> 
> The subprocess and its connections are set up with
> 
>   proc = subprocess.Popen(launchargs,stdin=subprocess.PIPE,
>     stdout=subprocess.PIPE, env=env)
> 
>   ...
>   self.reader = pickle.Unpickler(self.proc.stdout)
>   self.writer = pickle.Pickler(self.proc.stdin, 2)
> 
> after which I get
> 
>   result = self.reader.load()
> TypeError: 'str' does not support the buffer interface
> 
> That's as far as traceback goes, so I assume this is
> disappearing into C code.
> 
> OK, I know I need a byte stream.  I tried
> 
>   self.reader = pickle.Unpickler(self.proc.stdout.buffer)
>   self.writer = pickle.Pickler(self.proc.stdin.buffer, 2)
> 
> That's not allowed.  The "stdin" and "stdout" that are
> fields of "proc" do not have "buffer".  So I can't do that
> in the parent process.  In the child, though, where
> stdin and stdout come from "sys", "sys.stdin.buffer" is valid.
> That fixes the ""str" does not support the buffer interface
> error."  But now I get the pickle error "Ran out of input"
> on the process child side.  Probably because there's a
> str/bytes incompatibility somewhere.
> 
> So how do I get clean binary byte streams between parent
> and child process?

I don't know what you have to do to rule out deadlocks when you use pipes 
for both stdin and stdout, but binary streams are the default for 
subprocess. Can you provide a complete example?

Anyway, here is a demo for two-way communication using the communicate() 
method:

$ cat parent.py
import pickle
import subprocess

data = (5, 4.3, "üblich ähnlich nötig")

p = subprocess.Popen(
    ["python3", "child.py"],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE)

result = p.communicate(pickle.dumps(data, protocol=2))[0]
print(pickle.loads(result))

$ cat child.py
import sys
import pickle

a, b, c = pickle.load(sys.stdin.buffer)
pickle.dump((a, b, c.upper()), sys.stdout.buffer)

$ python3 parent.py 
(5, 4.3, 'ÜBLICH ÄHNLICH NÖTIG')

This is likely not what you want because here everything is buffered so that 
continuous interaction is not possible.