[Python-bugs-list] [ python-Bugs-520644 ] __slots__ are not pickled

noreply@sourceforge.net noreply@sourceforge.net
Thu, 14 Mar 2002 10:49:44 -0800


Bugs item #520644, was opened at 2002-02-20 20:50
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=520644&group_id=5470

Category: Type/class unification
>Group: Python 2.2.1 candidate
Status: Open
Resolution: None
Priority: 7
Submitted By: Samuele Pedroni (pedronis)
Assigned to: Guido van Rossum (gvanrossum)
Summary: __slots__ are not pickled

Initial Comment:
[Posted on behalf of Kevin Jacobs]

I have been hacking on ways to make lighter-weight 
Python objects using the
__slots__ mechanism that came with Python 2.2 new-
style class.  Everything
has gone swimmingly until I noticed that slots do not 
get pickled/cPickled
at all!

Here is a simple test case:

  import pickle,cPickle
  class Test(object):
    __slots__ = ['x']
    def __init__(self):
      self.x = 66666

  test = Test()

  pickle_str  = pickle.dumps( test )
  cpickle_str = cPickle.dumps( test )

  untest  = pickle.loads( pickle_str )
  untestc = cPickle.loads( cpickle_str )

  print untest.x    # raises AttributeError
  print untextc.x   # raises AttributeError

...

see 

http://aspn.activestate.com/ASPN/Mail/Message/python-
dev/1031499

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-07 02:43

Message:
Logged In: YES 
user_id=6380

I've decided on a "fix" that will go into 2.2.1.

When a new-style class defines slots and does not define
__getstate__, a dummy __getstate__ will be inserted into the
class that raises an exception. If the class wants slots to
be pickled, it should define a __getstate__ method.

In 2.3, we may revise this decision. The solution space is
large, and I don't want to rush it, hence the decision to
make it fail cleanly in 2.2.1 -- that's the best I can do.
The 2.2.1 solution will be compatible with whatever we
decide to do in 2.3.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-02-28 19:36

Message:
Logged In: YES 
user_id=6380

Good point. So maybe it should be up to the class to define
how to pickle slots.

An alternative idea could look at the type of descriptors;
slots use a different type than properties.

----------------------------------------------------------------------

Comment By: Samuele Pedroni (pedronis)
Date: 2002-02-28 19:29

Message:
Logged In: YES 
user_id=61408

[Guido on python-dev]
In particular, the fact that instances of classes with 
__slots__
appear picklable but lose all their slot values is a bug -- 
these
should either not be picklable unless you add a __reduce__ 
method, or
they should be pickled properly. ...
  I haven't made up my mind on how to fix this -- it would 
be
nice if __slots__ would automatically be pickled, but it's 
tricky
(although I think it's doable -- without ever referencing 
the
__slots__ variable :-).


[pedronis - my 2cts]
unless you plan some low-level (non-python-level) solution,
I think a main question is whether member and properties
are distinguishable and maybe whether among members
basic type members (file.softspace etc) and __slots__
members are distinguishable

It would be somehow strange and redundant if properties 
value would be automatically pickled (I see them as
computed value)

In java (bean) properties are not pickled and even
fields (= slots) can be marked as transient to avoid
their serialization.

In your picture it seems that all those things are
not to be dinstinguished, so probably no automatic
serialization, if there are members and given that
actually files for example cannot be pickled, would
be a reasonable solution.

Otherwise (distinguishable case) other 
automatic approaches can make sense too.

Just some - I hope valuable -  input and my opinion.

----------------------------------------------------------------------

Comment By: Kevin Jacobs (jacobs99)
Date: 2002-02-22 15:03

Message:
Logged In: YES 
user_id=459565

Oops.  Please ignore the last paragraph of point #5.
Samuele's __allslots__ is fine with regard to the
example I presented.

----------------------------------------------------------------------

Comment By: Kevin Jacobs (jacobs99)
Date: 2002-02-22 14:52

Message:
Logged In: YES 
user_id=459565

Samuele's sltattr.py is an interesting approach, though I 
am not entirely
sure it is necessary or feasible sufficiently address the 
significant
problems with slots via proxying __dict__ (see #5 below).  
Here is a mostly
complete list of smaller changes that are somewhat 
orthogonal to how we
address accesses to __dict__:

  1) Flatten slot lists:  Change obj.__class__.__slots__ to 
return an
     immutable list of all slot descriptors in the object 
(including all
     those of base classes).  The motivation for this is 
similar in spirit
     to storing a flattened __mro__.

     The advantages of this change are:

     a) allows for fast and explicit object reflection that 
correctly finds
        all dict attributes, all slot attributes.

     b) allows reflection implementations (like vars
(object) and pickle) to
        treat dict and slot attrs differently if we choose 
not to proxy
        __dict__.  This has several advantages, as 
explained in change #2.
        Also importantly, this way it is not possible 
to "lose" descriptors
        permanently by deleting them from 
obj.__class__.__dict__.

  2) Update reflection API even if we do not choose to 
proxy __dict__: Alter
     vars(object) to return a dictionary of all attributes, 
including both
     the contents of the non-proxied __dict__ and the valid 
attributes that
     result from iterating over __slots__ and evaluating 
the descriptors.
     The details of how this is best implemented depend on 
how we wish to
     define the behavior of modifying the resulting 
dictionary.  It could be
     either:

           a) explicitly immutable, which involves creating 
proxy objects
           b) mutable, which involves copying
           c) undefined, which means implicitly immutable

     Aside from the questions over the nature of the return 
type, this
     implementation (coupled with #1) has distinct 
advantages.  Specifically
     the native object.__dict__ has a very natural internal 
representation
     that pairs attribute names directly with values.  In 
contrast, a fair
     amount of additional work is needed to extract the 
slots that store
     values and create a dictionary of their names and 
values.  Other
     implementations will require a great deal more work 
since they would
     have to traverse though base classes to collecting 
slot descriptors.

  3) Flatten slot inheritance:  Update the new-style object 
inheritance
     mechanism to re-use slots of the same name, rather 
than creating a new
     slot and hiding the old.  This makes the inheritance 
semantics of slots
     equivalent to those of normal instance attributes and 
avoids
     introducing an ad-hoc and obscure method of data 
hiding.

  4) Update standard library to use new reflection API (and 
make them robust
     to properies at the same time) if we choose not to 
proxy __dict__.
     Virtually all of the changes are simple and involve 
updating these
     constructs:

           a) obj.__dict__
           b) obj.__dict__[blah]
           c) obj.__dict__[blah] = x

     (What these will become depends on other factors, 
including the context
      and semantics of vars(obj).)

     Here is a fairly complete list of Python 2.2 modules 
that will need to
     be updated:

       copy, copy_reg, inspect, pickle, pydoc, cPickle, 
Bastion, codeop,
       dis, doctest, gettext, ihooks, imputil, knee, pdb, 
profile, rexec,
       rlcompleter, tempfile, unittest, xmllib, xmlrpclib

    5) (NB: potentially controversial and not required) We 
could alter the
       descriptor protocol to make slots (and properties) 
more transparent
       when the values they reference do not exist.  Here 
is an example to
       illustrate this:

           class A(object):
             foo = 1

           class B(A):
             __slots__ = ('foo',)

           b = B()
           print b.foo
           > 1 or AttributeError?

        Currently an AttributeError is raised.  However, it 
is a fairly easy
        change to make AttributeErrors signal that 
attribute resolution is
        to continue until either a valid descriptor is 
evaluated, an
        instance-attribute is found, or until the 
resolution fails after
        search the meta-type, the type and the instance 
dictionary.

        The problem illustrated by the above code also 
occurs when trying to
        create proxies for __dict__, if the proxy worked on 
the basis of the
        collected slot descriptors (__allslots__ in 
Samuele's example).

I am prepared to submit patches to address each of these 
issues.  However, I
do want feedback beforehand, so that I do not waste time 
implementing
something that will never be accepted.


----------------------------------------------------------------------

Comment By: Samuele Pedroni (pedronis)
Date: 2002-02-22 01:33

Message:
Logged In: YES 
user_id=61408

some slots more like attrs illustrative python code

----------------------------------------------------------------------

Comment By: Kevin Jacobs (jacobs99)
Date: 2002-02-21 17:51

Message:
Logged In: YES 
user_id=459565

This bug raises questions about what a slot really is.
After a fair amount of discussion on Python-dev, we
have come up with basically two answers:

  1) a slot is a struct-member that is part of the
     private implementation of an object.  Slots should
     have their own semantics and not be expected to
     act like Python instance attributes.

  2) slots should be treated just like dict instance
     attributes except they are allocated statically
     within the object itself, and require slightly 
     different reflection methods.

Under (1), this bug isn't really a bug.  The class should
implement a __reduce__ function or otherwise hook into
the copy_reg system.

Under (2), this bug is just the tip of the iceberg. 
There are about 8 other problems with the current 
slot implementation that need to be resolved before
slots act almost identically to normal instance attributes.
Thankfully, I am fairly confident that I can supply 
patches that can achieve this, though I am waiting for
Guido to comment on this issue when he returns from his
trip.


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=520644&group_id=5470