[Python-Dev] [Python 2.2 BUG] pickle/cPickle does not find slots

Fri, 15 Feb 2002 09:57:12 -0800 (PST)

Hi Kevin.  I'm with you slots have great potential. 
First of all, I think it is a very bad idea that slots
could be mutable.  Here is something I wrote on the
python-list.  (Python list is too noisy and too many
dumb questions involving people not thinking before
tackle a problem. -with the exception of an
interesting thread on licensing, and a spanish
message.)

Anyway, here is some food for thought on slots.
Perhaps long-winded and already talked about, but here
it is:

(If you want to skip the intro and get to the meat and
potatos go to the code, and read my remarks.

- - - - - -
Hello Python developers!  I have been reading about
Python 2.2's new features and I have to say there's
lots of good stuff going on. The feature I dedicate
this message to is slots.  This ubiquitous new feature
has the potential to boost performance of python quite
considerable and it is the first time python has ever
hinted at, for lack of a better word, "being static".

I used this word loosely.  What I mean by this is that
typically one could set attributes arbitrarily on
python objects, but now cometh slots, if __slots__ is
defined, one could set only attribute names defined in
slots if __slots__ exists.

In introductory essay on the new python object system,
"Unifying types and classes in 2_2", I found the
benevolent dictator's answer for the existence of
__slots__ unsatisfactory (or at least the example). 
As implied from the introductory essay, __slots__ was
used as a means to eliminate a double dictionary for
the dictionary subclass in the example (please don't
be offended -- SO what).  That is one application of
slots, but there is a more powerful way to use this
new
feature.

First let me discuss some notational issues I have
with __slots__.  It is possible to use any object that
supports iteration as the __slots__ object (huh does
my memory serve me correctly?). I feel that this is
not a good idea.  Consider the following example:

    class Foo(object):
        __slots__=['a','b','c']

    foo=Foo()

Bad, but you could do it.

    foo.__class__.__slots__.append('d') #Ouch!

Suppose someone wrote code to initialize some instance
like so:

    for attr in foo.__class__.__slots__:
        setattr(foo,attr,0)

Granted this is decidedly bad code but the problem
this creates is that attribute "d" does not exist in
foo or any instance of Foo before or after the "append
event".  So an attribute error will be raised. Slot
attributes are read only thus del foo.__slots__ is not
possible.  So neither should __slots__.append be
legal.  So the answer is to enforce __slots__ is
tuple, or for a single attribute __slots__ is string,
when compiling objects.

Another issue I have with __slots__ is that there
exists better notation to do the same thing.  And part
of python is to maintain a clean coding style. Why not
introduce new a keyword into the language so
"static attribute sets" are created like so:

    class Foo(object):
        attribute a
        attribute b
        attribute c

Perhaps this syntax could be expanded further to
include strict typing:

    class Foo(object):
        attribute a is str
        attribute b is float
        attribute c is int

I feel that this notation is much more lucid and could
lead to even more interesting notation:

    class Foo(object):
	attribute a is str
	attribute b is float
	attribute c is property
	attribute g is property of float

I hope that it is understood from this context that
"is" compares the type(<attrname>) to trailing type in
the declaration.  The meaning of the modifier "of" in
the last line is a bit ambiguous at this moment. 
Perhaps it enforces the type passed into fset, and the
return value of fget.

The programmer doesn't need to know what the
underlying __slots__ structure looks like, unless they
are doing some heavy duty C.  In which case they could
probably handle it in all it's grand ugliness.

Is the python interpreter  __slot__ aware?  Instead of
the usual variable name lookup into __dict__ (which is
not supported by instances who's class implements
__slots__), a slot attribute could be directly
referenced via pointer indirection.   Does the
interpreter substitute references for slot
attributes?  For that matter any namespace could
utilize this new approach, even the local
namespace of a code segment. Slots could be built
implicitly for a local namespace.  This could lead to
significant performance boost. Or perhaps a JIT of
sorts for python. Cause all you need is a bit of type
info, and those segments could be accurately
translated into native code.
------

--- Kevin Jacobs <jacobs@penguin.theopalgroup.com>
wrote:
> [I tried to post this on SourceForge, but as usual,
> it hates my guts]
> 
> I have been hacking on ways to make lighter-weight
> Python objects using the
> __slots__ mechanism that came with Python 2.2
> new-style class.  Everything
> has gone swimmingly until I noticed that slots do
> not get pickled/cPickled
> at all!
> 
> Here is a simple test case:
> 
>   import pickle,cPickle
>   class Test(object):
>     __slots__ = ['x']
>     def __init__(self):
>       self.x = 66666
> 
>   test = Test()
> 
>   pickle_str  = pickle.dumps( test )
>   cpickle_str = cPickle.dumps( test )
> 
>   untest  = pickle.loads( pickle_str )
>   untestc = cPickle.loads( cpickle_str )
> 
>   print untest.x    # raises AttributeError
>   print untextc.x   # raises AttributeError
> 
> Clearly, this is incorrect behavior.  The problem is
> due a change in object
> reflection semantics.  Previously (before type-class
> unification), a
> standard Python object instance always contained a
> __dict__ that listed all
> of its attributes.  Now, with __slots__, some
> classes that do store
> attributes have no __dict__ or one that only
> contains what did not fit into
> slots.
> 
> Unfortunately, there is no trivial way to know what
> slots a particular class
> instance really has. This is because the __slots__
> list in classes and
> instances can be mutable!  Changing these lists
> _does not_ change the object
> layout at all, so I am unsure why they are not
> stored as tuples and the
> '__slots__' attribute is not made read-only.  To be
> pedantic, the C
> implementation does have an immutable and canonical
> list(s) of slots, though
> they are well buried within the C extended type
> implementation.
> 
> So, IMHO this bug needs to be fixed in two steps:
> 
> First, I propose that class and instance __slots__
> read-only and the lists
> made immutable.  Otherwise, pickle, cPickle, and any
> others that want to use
> reflection will be SOL.  There is certainly good
> precedent in several places
> for this change (e.g., __bases__, __mro__, etc.) I
> can submit a fairly
> trivial patch to do so.  This change requires
> Guido's input, since I am
> guessing that I am simply not privy to the method,
> only the madness.
> 
> Second, after the first issue is resolved, pickle
> and cPickle must then be
> modified to iterate over an instance's __slots__ (or
> possibly its class's)
> and store any attributes that exist.  i.e., some
> __slots__ can be empty and
> thus should not be pickled.  I can also whip up
> patches for this, though I'll
> wait to see how the first issue shakes out.
> 
> Regards,
> -Kevin
> 
> PS:  There may be another problem when when one
> class inherits from another
>      and both have a slot with the same name.
> 
>      e.g.:
>        class Test(object):
>          __slots__ = ['a']
> 
>        class Test2(Test):
>          __slots__ = ['a']
> 
>        test=Test()
>        test2=Test2()
>        test2.__class__ = Test
> 
>     This code results in this error:
> 
>       Traceback (most recent call last):
>         File "<stdin>", line 1, in ?
>       TypeError: __class__ assignment: 'Test' object
> layout differs from 'Test2'
> 
>     However, Test2's slot 'a' entirely masks Test's
> slot 'a'.  So, either
>     there should be some complex attribute access
> scheme to make both slots
>     available OR (in my mind, the preferable
> solution) slots with the same
>     name can simply be re-used or coalesced.  Now
> that I think about it,
>     this has implications for pickling objects as
> well.  I'll likely leave
>     this patch for Guido -- it tickles some fairly
> hairy bits of typeobject.
> 
>     Cool stuff, but the rabbit hole just keeps
> getting deeper and deeper....
> 
> --
> Kevin Jacobs
> The OPAL Group - Enterprise Systems Architect
> Voice: (216) 986-0710 x 19         E-mail:
> jacobs@theopalgroup.com
> Fax:   (216) 986-0714              WWW:   
> http://www.theopalgroup.com
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev

__________________________________________________
Do You Yahoo!?
Got something to say? Say it better with Yahoo! Video Mail 
http://mail.yahoo.com

[Python-Dev] [Python 2.2 BUG] pickle/cPickle does not find __slots__

[Python-Dev] [Python 2.2 BUG] pickle/cPickle does not find slots