in a pickle

[David Raymond]

https://docs.python.org/3.7/library/pickle.html#pickling-class-instances includes 2 notes, which I think are the relevant ones:

When a class instance is unpickled, its __init__() method is usually not invoked. The default behaviour first creates an uninitialized instance and then restores the saved attributes.

At unpickling time, some methods like __getattr__(), __getattribute__(), or __setattr__() may be called upon the instance. In case those methods rely on some internal invariant being true, the type should implement __getnewargs__() or __getnewargs_ex__() to establish such an invariant; otherwise, neither __new__() nor __init__() will be called.


So I think that without using the various dunders you can't rely on the bits being restored in a specific order.

So at the point in the restoration where you're getting the exception, self.shape hasn't been restored yet while it's trying to set self[1], which is why trying to print self.shape is failing. Here's a modified version with more prints to show some of that:


import pickle
class test(dict):
    def __init__(self, keys, shape = None):
        print("__init__ is running")
        print("keys:", keys)
        print("shape:", shape)
        
        self.shape = shape
        for key in keys:
            self[key] = None
    
    def __setitem__(self, key, val):
        print("__setitem__ is running")
        print("self:", self)
        print("key:", key)
        print("val:", val)
        
        try:
            print("self.shape:", self.shape)
        except AttributeError as err:
            print("AttributeError:", err)
        
        dict.__setitem__(self, key, val)

x = test([1, 2, 3])
print(x, x.shape)
s = pickle.dumps(x)
print("\n---About to load it---\n")
y = pickle.loads(s)
print(".loads() complete")
print(y, y.shape)



This is what that outputs:


__init__ is running
keys: [1, 2, 3]
shape: None
__setitem__ is running
self: {}
key: 1
val: None
self.shape: None
__setitem__ is running
self: {1: None}
key: 2
val: None
self.shape: None
__setitem__ is running
self: {1: None, 2: None}
key: 3
val: None
self.shape: None
{1: None, 2: None, 3: None} None

---About to load it---

__setitem__ is running
self: {}
key: 1
val: None
AttributeError: 'test' object has no attribute 'shape'
__setitem__ is running
self: {1: None}
key: 2
val: None
AttributeError: 'test' object has no attribute 'shape'
__setitem__ is running
self: {1: None, 2: None}
key: 3
val: None
AttributeError: 'test' object has no attribute 'shape'
.loads() complete
{1: None, 2: None, 3: None} None



Alas, I can't offer any help with how to use __getnewargs__() or the other dunders to properly handle it.



-----Original Message-----
From: Python-list [mailto:python-list-bounces+david.raymond=tomtom.com at python.org] On Behalf Of duncan smith
Sent: Wednesday, March 06, 2019 11:14 AM
To: python-list at python.org
Subject: in a pickle

Hello,
      I've been trying to figure out why one of my classes can be
pickled but not unpickled. (I realise the problem is probably with the
pickling, but I get the error when I attempt to unpickle.)

A relatively minimal example is pasted below.


>>> import pickle
>>> class test(dict):
	def __init__(self, keys, shape=None):
		self.shape = shape
		for key in keys:
			self[key] = None

	def __setitem__(self, key, val):
		print (self.shape)
		dict.__setitem__(self, key, val)

		
>>> x = test([1,2,3])
None
None
None
>>> s = pickle.dumps(x)
>>> y = pickle.loads(s)
Traceback (most recent call last):
  File "<pyshell#114>", line 1, in <module>
    y = pickle.loads(s)
  File "<pyshell#111>", line 8, in __setitem__
    print (self.shape)
AttributeError: 'test' object has no attribute 'shape'


I have DUCkDuckGo'ed the issue and have tinkered with __getnewargs__ and
__getnewargs_ex__ without being able to figure out exactly what's going
on. If I comment out the print call, then it seems to be fine. I'd
appreciate any pointers to the underlying problem. I have one or two
other things I can do to try to isolate the issue further, but I think
the example is perhaps small enough that someone in the know could spot
the problem at a glance. Cheers.

Duncan
-- 
https://mail.python.org/mailman/listinfo/python-list