pickle.py and cPickle.c - persistent_id is always called - why?

[matsaleh at my-deja.com]

Hello fellow .py types...

I am doing a bit of work using pickle/cpickle and
am trying to optimize. I found that my user-
defined persistent_id() function is being called
for (just about) every attribute (name and value)
in my object that is being pickled.

My specific (somewhat ad-hoc) test case pickles
~100 objects in a containment hierarchy, using
persistent_id to break the containment
relationship and replace the references with a
proprietary object id. Although I have only ~100
object references to resolve, persistent_id() is
called ~6400 times. In tracing the code, it
appears to be called for every attribute and
value in my objects.

This appears to be caused by the fact that
pickle.py:save() is called with the default
pers_save flag of 0 in all cases except for when
it is called by save_pers(). This appears to
indicate that all types: tuples, dicts,
sequences, longs, strings, etc, cause my
persistent_id() to be called, when all I want is
for it to be called for object instances.

I modified pickle.py to change the default of the
pers_save flag to 1 in the save() method, and
then call it with a 0 only from within the
save_inst() method. This amounts to invoking my
persistent_id() function only when a reference to
an object instance is encountered as my objects
are being pickled.

My pickled objects do not appear to be adversely
affected by this change, and the number of calls
to persistent_id() was reduced from ~6400 to
~500, reducing the time spent in this method by
an order of magnitude (0.37 sec to 0.03 sec).

I have not yet tested this change in cPickle.c,
but in my original tests, persistent_id() was a
much more significant factor in my runs using
cPickle, because all the pickling code is in C
and is relatively much faster than my
persistent_id() method, which is in Python. I
expect the performance boost my making the change
in cPickle to be even greater, relatively
speaking.

My question is, why is persistent_id() being
invoked so often? I do not see the reason for
calling it for basic python types and structures
such as tuples, dicts, and the like. Is this a
reasonable change to make to the python source,
or is it not a safe change for the general
pickling cases that the pickle/cPickle modules
have to handle? I'm sure this code has been
scruitinized by many folks much more experienced
than I - I would be grateful for your insights.

Regards.



Sent via Deja.com http://www.deja.com/
Before you buy.