[Python-Dev] Pickler/Unpickler API clarification

[Collin Winter]

I'm working on some performance patches for cPickle, and one of the
bigger wins so far has been replacing the Pickler's memo dict with a
custom hashtable (and hence removing memo's getters and setters). In
looking over this, Jeffrey Yasskin commented that this would break
anyone who was accessing the memo attribute.

I've found a few examples of code using the memo attribute ([1], [2],
[3]), and there are probably more out there, but the memo attribute
doesn't look like part of the API to me. It's only documented in
http://docs.python.org/library/pickle.html as "you used to need this
before Python 2.3, but don't anymore". However: I don't believe you
should ever need this attribute.

The usages of memo I've seen break down into two camps: clearing the
memo, and wanting to explicitly populate the memo with predefined
values. Clearing the memo is recommended as part of reusing Pickler
objects, but I can't fathom when you would want to reuse a Pickler
*without* clearing the memo. Reusing the Pickler without clearing the
memo will produce pickles that are, as best I can see, invalid -- at
least, pickletools.dis() rejects this, which is the closest thing we
have to a validator. Explicitly setting memo values has the same
problem: an easy, very brittle way to produce invalid data.

So my questions are these:
1) Should Pickler/Unpickler objects automatically clear their memos
when dumping/loading?
2) Is memo an intentionally exposed, supported part of the
Pickler/Unpickler API, despite the lack of documentation and tests?

Thanks,
Collin

[1] - http://google.com/codesearch/p?hl=en#Qx8E-7HUBTk/trunk/google/appengine/api/memcache/__init__.py&q=lang:py%20%5C.memo
[2] - http://google.com/codesearch/p?hl=en#M-DDI-lCOgE/lib/python2.4/site-packages/cvs2svn_lib/primed_pickle.py&q=lang:py%20%5C.memo
[3] - http://google.com/codesearch/p?hl=en#l_w_cA4dKMY/AtlasAnalysis/2.0.3-LST-1/PhysicsAnalysis/PyAnalysis/PyAnalysisUtils/python/root_pickle.py&q=lang:py%20pick.*%5C.memo%5Cb