Using "pickle" for interprocess communication - some notes and things that ought to be documented.

[John Nagle]

Carl Banks wrote:
> On Jan 17, 2:28 pm, John Nagle <na... at animats.com> wrote:

> 
>> It's also necessary to call Pickle's "clear_memo" before each "dump"
>> call, since objects might change between successive "dump" calls.
>> "Unpickle" doesn't have a "clear_memo" function.  It should, because
>> if you keep reusing the "Unpickle" object, the memo dictionary
>> fills up with old objects which can't be garbage collected.
>> This creates a memory leak in long-running programs.
> 
> This is all good to know.  I agree that this is a good use case for a
> clear_memo on a pickle unloader.

	reader = pickle.Unpickler(self.datain)	# set up reader
	....
	reader.memo = {}		# no memory from cycle to cycle

> 
> 
>> Then, on Windows, there's a CR LF problem. This can be fixed by
>> launching the subprocess with
>>
>>     proc = subprocess.Popen(launchargs,
>>         stdin=subprocess.PIPE, stdout=subprocess.PIPE,
>>         universal_newlines=True)
>>
>> Failure to do this produces the useful error message "Insecure string pickle".
>> Binary "pickle" protocol modes won't work at all in this situation; "universal
>> newline" translation is compatible, not transparent.  On Unix/Linux, this
>> just works, but the code isn't portable.
> 
> I would think a better solution would be to use the -u switch to
> launch the subprocess, or the PYTHONUNBUFFERED environment variable if
> you want to invoke the Python script directly.  It opens up stdin and
> stdout in binary, unbuffered mode.

    Ah.  That works.  I wasn't aware that "unbuffered" mode also implied
binary transparency.  I did that, and now cPickle works in both text (0)
and binary (2) protocol modes.  Turned off "Universal Newline" mode.

> Thanks: this was an informative post

    Thanks.  We have this working well now.  After a while, I'll publish
the module, which is called "subprocesscall.py".

				John Nagle