AssertionError in pickle's memoize function

[Michael Hohn]

Tim Peters <tim.peters at gmail.com> writes:

> [Michael Hohn]
> > under python 2.2, the pickle/unpickle sequence incorrectly restores
> > a larger data structure I have.
> >
> > Under Python 2.3, these structures now give an explicit exception from
> > Pickle.memoize():
> >    assert id(obj) not in self.memo
> > 
> > I'm shrinking the offending data structure down to find the problem
> > and provide an easily reproducible example,
> > but maybe someone on the list could tell me under what general
> > conditions this assertion is expected to fail.
> 
> Assertions are never expected to fail, so "something impossible
> happened" when they do fail.
> 
> See whether your Python has Lib/pickletools.py.  There's an enormous
> amount of info about pickles in that (for example, it will tell you
> what "memo" means).
> 
> May help to try cPickle instead of pickle.  Since they're distinct
> implementations, they have different bugs.  cPickle can be much faster
> than pickle, but it's a lot easier to understand pickle.py.

Here is a code sample that shows the problem I ran into:

test.py:
=================================
import pickle

class aList(list):
    def __init__(self, arg):
        # w/o this call, pickle works...
        list.__init__(self, arg)
        pass

A = aList([1,2])
B = aList([A, 3])

the_data = {'a': A, 'b': B}
A._stored_by = the_data

pickle.dumps([the_data, B])             # ok
pickle.dumps([B, the_data])             # fails

=================================


Outputs under:

Python 2.3 (#1, Sep 13 2003, 00:49:11) 
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin


9 scarlet::~:0> python test.py 
Traceback (most recent call last):
  File "test.py", line 16, in ?
    pickle.dumps([B, the_data])             # fails
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 1386, in dumps
    Pickler(file, protocol, bin).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 231, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 293, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 614, in save_list
    self._batch_appends(iter(obj))
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 629, in _batch_appends
    save(x)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 338, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 419, in save_reduce
    self.memoize(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p
ickle.py", line 251, in memoize
    assert id(obj) not in self.memo
AssertionError

with the same problem under python on linux:

Python 2.3 (#1, Jul 31 2003, 14:19:24) 
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2


Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/tmp/python-286703ll", line 1, in ?
    pickle.dumps([B, the_data])             # fails
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 1386, in dumps
    Pickler(file, protocol, bin).dump(obj)
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 231, in dump
    self.save(obj)
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 293, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 614, in save_list
    self._batch_appends(iter(obj))
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 629, in _batch_appends
    save(x)
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 338, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 419, in save_reduce
    self.memoize(obj)
  File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 251, in memoize
    assert id(obj) not in self.memo
AssertionError