Using "pickle" for interprocess communication - some notes and things that ought to be documented.

[John Nagle]

It's possible to use "pickle" for interprocess communication over
pipes, but it's not straightforward.

First, "pickle" output is self-delimiting.
Each dump ends with ".", and, importantly, "load" doesn't read
any characters after the "."  So "pickle" can be used repeatedly
on the same pipe, and one can do repeated message-passing this way.  This
is a useful, but undocumented, feature.

It almost works.

Pickle's "dump" function doesn't flush output after dumping, so
there's still some data left to be written.  The sender has to
flush the underlying output stream after each call to "dump",
or the receiver will stall. The "dump" function probably ought to flush
its output file.

It's also necessary to call Pickle's "clear_memo" before each "dump"
call, since objects might change between successive "dump" calls.
"Unpickle" doesn't have a "clear_memo" function.  It should, because
if you keep reusing the "Unpickle" object, the memo dictionary
fills up with old objects which can't be garbage collected.
This creates a memory leak in long-running programs.

Then, on Windows, there's a CR LF problem. This can be fixed by
launching the subprocess with

    proc = subprocess.Popen(launchargs,
	stdin=subprocess.PIPE, stdout=subprocess.PIPE,
	universal_newlines=True)	

Failure to do this produces the useful error message "Insecure string pickle".
Binary "pickle" protocol modes won't work at all in this situation; "universal
newline" translation is compatible, not transparent.  On Unix/Linux, this
just works, but the code isn't portable.

Incidentally, in the subprocess, it's useful to do

	sys.stdout = sys.stderr

after setting up the Pickle objects.  This prevents any stray print statements
from interfering with the structured Pickle output.

Then there's end of file detection.  When "load" reaches an end of
file, it properly raises EOFError.  So it's OK to do "load" after
"load" until EOFerror is raised.

"pickle" and "cPickle" seem to be interchangeable in this application,
so that works.

It's a useful way to talk to a subprocess, but you need to know all the
issues above to make it work.

				John Nagle