New classes, __slots__, __dict__

[David Mertz, Ph.D.]

Hi Pythonistas,

This is especially to the actual Python developers, but other users
might have insight.  For background, I am co-author of the module
xml_pickle; for those unfamiliar, xml_pickle (more-or-less) does a
superset of what pickle/cPickle do, but using XML files as the data
format.  This note isn't to plug the module, but you can find info at:

    http://gnosis.cx/download/Gnosis_XML_Util.ANNOUNCE

In xml_pickle, we are trying to decide how to handle Python 2.2+ new
style classes that use the '__slots__' attribute.  Unfortunately,
pickle.py itself doesn't really do anything very sensible for
guidance... which is the nature of my question.

Let me explain for those (vast majority) of readers who have not
encountered this.  The Python 2.2(+.1c1) release notes indicate:

  - If you try to pickle an instance of a class that has __slots__ but
    doesn't define or override __getstate__, a TypeError is now raised.
    This is done by adding a bozo __getstate__ to the class that always
    raises TypeError.  (Before, this would appear to be pickled, but the
    state of the slots would be lost.)

I still have plain 2.2 on my system, so what I get is:

    >>> class New(object):
    ...     __slots__ = ('this','that')
    ...
    >>> new = New()
    >>> new.this = 'this'
    >>> from pickle import dumps, loads
    >>> new2 = loads(dumps(new))
    >>> new2.this
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    AttributeError: this

But it's easy enough to envision a TypeError being raised here instead.
In any case, neither behavior is particularly helpful.

I'd like pickle to do something nicer, but MY real concern is making
xml_pickle do something nice.  Within xml_pickle, we look at
"o.__dict__.keys()" to find the attributes for an object we are
pickling.  In an old version, xml_pickle looked at "dir(o)", but that
was fragile, and broke with Python 2.2.  The o.__dict__ approach doesn't
work with new style classes that have __slots__:

    >>> new.__dict__
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    AttributeError: 'New' object has no attribute '__dict__'

My feeling on this is that we (the xml_pickle authors) should provide a
"smart" attibute detector that looks something like:

    def dir_(o):
        if hasattr(o,'__dict__'):
            return o.__dict__.keys()
        if not hasattr(o,'__slots__'):
            raise TypeError, "Strange class with neither dict nor slots"
        attrs = []
        for attr in o.__slots__:
            if hasattr(o, attr):
                attrs.append(attr)
        return attrs

At first brush, this function returns helpful results in all cases:

    >>> class Old: pass
    ...
    >>> class Noslots(object): pass
    ...
    >>> old = Old()
    >>> old.this = 'this'
    >>> noslots = Noslots()
    >>> noslots.this = 'this'
    >>> dir_(new), dir_(old), dir_(noslots)
    (['this'], ['this'], ['this'])

However, my xml_pickle co-author, Frank McIngvale, expressed the
following reservation about substituting my new dir_() function:

| Also, my admittedly simpleminded thought is that if handling slots
| is a 2-liner, why doesn't pickle.py just do that? I have to believe that
| the pickle authors are smart people and there is some deep technical
| reason why getstate MUST exist if slots are used, and not just that the
| pickle authors were too lazy and punted.

So what do you think... smart people who wrote pickle?  What's wrong
with the (slightly-more-than-two) line solution?  What is the deep
technical mojo that argues against this approach?

Yours, David...


--
    _/_/_/ THIS MESSAGE WAS BROUGHT TO YOU BY: Postmodern Enterprises _/_/_/
   _/_/    ~~~~~~~~~~~~~~~~~~~~[mertz at gnosis.cx]~~~~~~~~~~~~~~~~~~~~~  _/_/
  _/_/  The opinions expressed here must be those of my employer...   _/_/
 _/_/_/_/_/_/_/_/_/_/ Surely you don't think that *I* believe them!  _/_/