Python3 "pickle" vs. stdin/stdout - unable to get clean byte streams in Python 3

[Cameron Simpson]

On 12Mar2015 12:55, John Nagle <nagle at animats.com> wrote:
>  I have working code from Python 2 which uses "pickle"
>to talk to a subprocess via stdin/stdio.  I'm trying to
>make that work in Python 3.
>  First, the subprocess Python is invoked with the "-d' option,
>so stdin and stdio are supposed to be unbuffered binary streams.

You shouldn't need to use unbuffered streams specificly. It should be enough to 
.flush() the output stream (at whichever end) after you have written the pickle 
data.

I'm skipping some of your discussion; I can see nothing wong. I don't use 
pickle itself so aside from saying that your use seems to conform to the python 
3 doco I can't comment more deeply. That said, I do use subprocess a fair bit.

[...]
>  result = self.reader.load()
>TypeError: 'str' does not support the buffer interface
>That's as far as traceback goes, so I assume this is
>disappearing into C code.

No line numbers at all? Or, I suppose, just the line number from your program 
and nothing from the pickle module?

>OK, I know I need a byte stream.  I tried
>  self.reader = pickle.Unpickler(self.proc.stdout.buffer)
>  self.writer = pickle.Pickler(self.proc.stdin.buffer, 2)

You should not need to care about these. They're not required.

>That's not allowed.  The "stdin" and "stdout" that are
>fields of "proc" do not have "buffer".  So I can't do that
>in the parent process.  In the child, though, where
>stdin and stdout come from "sys", "sys.stdin.buffer" is valid.

But irrelevant. Besides, the stream buffer may not contain the whole pickle 
data anyway; it will be empty before a read and quite possibly incomplete 
afterwards. It is just a buffer.

>That fixes the ""str" does not support the buffer interface
>error."

I'm not sure "fix" is the right characterisation here.

>But now I get the pickle error "Ran out of input"
>on the process child side.  Probably because there's a
>str/bytes incompatibility somewhere.

No, probably because the buffer is only ever a snapshot of part of the stream.

str/bytes errors are more glaringly obviously so.

>So how do I get clean binary byte streams between parent
>and child process?

This is where I'm confused: my experience is that Popen.subprocess gives you 
binary streams; I always need to put an encoder/decoder on them to use text.  
Did that just the other day.

BTW, this is on some UNIX variant? Should not be very relevant...

Further questions:

What does self.proc.stdout.__class__ say? And for stdin?

Cheers,
Cameron Simpson <cs at zip.com.au>

My opinions are borrowed from someone who no longer needs them.
        -- KatmanDu at uga.cc.uga.edu