Shelve doesn't appear to restore references properly????

[Carl Banks]

Muppet wrote:
>> Nevertheless, you know your code better than I.  If pickling a dict
>> won't do, then you can work around the limitation of shelve using the
>> __getstate__ and __setstate__ methods.  I will be happy to elaborate
>> if you wish.
> 
> Please. I've spent most of the evening mucking around with
> __getstate__ and __setstate__ and the best I've been able to manage is
> a "borg" object, where the __dict__ attributes of the various restored
> objects are the same, which isn't really progress. I just can't figure
> out how to actually get shared objects back from the pickle, no matter
> how I try.
> 
> This is really driving me nuts!


Here's an example.  Consider the following class that defines
__getstate__ and __setstate__:

    class A:
        def __init__(self,data):
            self.shared_data = data
            # init rest of self
        def __getstate__(self):
            d = self.__dict__.copy()
            del d['shared_data']
            return d
        def __setstate__(self,state):
            self.__dict__.update(state)
            self.shared_data = set_shared_data_somehow()

Notice what __getstate__ does: it makes a copy of it's dict, and then
deletes the shared data from that dict.  This is so the shared data
doesn't get packaged into every shelf item.

Unfortunately, this burdens __setstate__ with the task of getting the
shared data back into its proper attribute.  How you accomplish this
depends on the nature of your shared data.

If there's only one shared object, you probably want to store it in
its own pickle file, or in as special entry in the shelf.  When you
unshelve the data, you'll want to retrieve the shared object first,
and store it in a global binding (say, The_Shared_Object).  Then, have
__setstate__ restore the link.  (Of course, if this is the case, it
raises the question of why you need your little muppets to reference
to this object in the first place.)

If there are many possible shared objects, then it might be better to
store all the shared data in another shelf.  Suppose the shelf called
"shared_shelf" stores all shared data.  Then, you could have
__getstate__ replace self.shared_data with a key in shared_shelf.
When __setstate__ is called, it can retrieve the object from that
shelf:

    class A:
        def __init__(self,data):
            self.shared_data = data
            # init rest of self
        def __getstate__(self):
            d = self.__dict__.copy()
            d['shared_data'] = self.shared_data.key
            return d
        def __setstate__(self,state):
            self.__dict__.update(state)
            self.shared_data = shared_shelf[self.shared_data]

Then, you have to be sure you properly stored the shared objects in
the shelves.

I hope this gives you a nudge in the right direction.  In general,
preserving complex shared data is not an easy task.  My final advice
is: think about ways to reorganize the data structures so that you
don't have to use shared objects, and maybe you can avoid this whole
mess.


-- 
CARL BANKS