in a pickle

[Peter Otten]

duncan smith wrote:

> On 06/03/2019 16:14, duncan smith wrote:
>> Hello,
>>       I've been trying to figure out why one of my classes can be
>> pickled but not unpickled. (I realise the problem is probably with the
>> pickling, but I get the error when I attempt to unpickle.)
>> 
>> A relatively minimal example is pasted below.
>> 
>> 
>>>>> import pickle
>>>>> class test(dict):
>> def __init__(self, keys, shape=None):
>> self.shape = shape
>> for key in keys:
>> self[key] = None
>> 
>> def __setitem__(self, key, val):
>> print (self.shape)
>> dict.__setitem__(self, key, val)
>> 
>> 
>>>>> x = test([1,2,3])
>> None
>> None
>> None
>>>>> s = pickle.dumps(x)
>>>>> y = pickle.loads(s)
>> Traceback (most recent call last):
>>   File "<pyshell#114>", line 1, in <module>
>>     y = pickle.loads(s)
>>   File "<pyshell#111>", line 8, in __setitem__
>>     print (self.shape)
>> AttributeError: 'test' object has no attribute 'shape'
>> 
>> 
>> I have DUCkDuckGo'ed the issue and have tinkered with __getnewargs__ and
>> __getnewargs_ex__ without being able to figure out exactly what's going
>> on. If I comment out the print call, then it seems to be fine. I'd
>> appreciate any pointers to the underlying problem. I have one or two
>> other things I can do to try to isolate the issue further, but I think
>> the example is perhaps small enough that someone in the know could spot
>> the problem at a glance. Cheers.
>> 
>> Duncan
>> 
> 
> OK, this seems to  be a "won't fix" bug dating back to 2003
> (https://bugs.python.org/issue826897). The workaround,
> 
> 
> class DictPlus(dict):
>   def __init__(self, *args, **kwargs):
>     self.extra_thing = ExtraThingClass()
>     dict.__init__(self, *args, **kwargs)
>   def __setitem__(self, k, v):
>     try:
>       do_something_with(self.extra_thing, k, v)
>     except AttributeError:
>       self.extra_thing = ExtraThingClass()
>       do_something_with(self.extra_thing, k, v)
>     dict.__setitem__(self, k, v)
>   def __setstate__(self, adict):
>     pass
> 
> 
> doesn't work around the problem for me because I need the actual value
> of self.shape from the original instance. But I only need it for sanity
> checking, and under the assumption that the original instance was valid,
> I don't need to do this when unpickling. I haven't managed to find a
> workaround that exploits that (yet?). Cheers.

I've been playing around with __getnewargs__(), and it looks like you can 
get it to work with a custom __new__(). Just set the shape attribute there 
rather than in __init__():

$ cat pickle_dict_subclass.py 
import pickle


class A(dict):
    def __new__(cls, keys=(), shape=None):
        obj = dict.__new__(cls)
        obj.shape = shape
        return obj

    def __init__(self, keys=(), shape=None):
        print("INIT")
        for key in keys:
            self[key] = None
        print("EXIT")

    def __setitem__(self, key, val):
        print(self.shape, ": ", key, " <-- ", val, sep="")
        super().__setitem__(key, val)

    def __getnewargs__(self):
        print("GETNEWARGS")
        return ("xyz", self.shape)

x = A([1, 2, 3], shape="SHAPE")
x["foo"] = "bar"
print("pickling:")
s = pickle.dumps(x)
print("unpickling:")
y = pickle.loads(s)
print(y)
$ python3 pickle_dict_subclass.py 
INIT
SHAPE: 1 <-- None
SHAPE: 2 <-- None
SHAPE: 3 <-- None
EXIT
SHAPE: foo <-- bar
pickling:
GETNEWARGS
unpickling:
SHAPE: 1 <-- None
SHAPE: 2 <-- None
SHAPE: 3 <-- None
SHAPE: foo <-- bar
{1: None, 2: None, 3: None, 'foo': 'bar'}

It's not clear to me how the dict items survive when they are not included 
in the __getnewargs__() result, but apparently they do.