[Tutor] BadPickleGet error

[Peter Otten]

Rick Pasotto wrote:

> Traceback (most recent call last):
>   File "/usr/share/rss2email/rss2email.py", line 748, in <module>
>     else: run()
>   File "/usr/share/rss2email/rss2email.py", line 488, in run
>     feeds, feedfileObject = load()
>   File "/usr/share/rss2email/rss2email.py", line 439, in load
>     feeds = pickle.load(feedfileObject)
> cPickle.BadPickleGet: 2574'8
> 
> Could someone please explain what the number(s) after 'BadPickleGet'
> mean?

It's a key into a dictionary with previously retrieved objects.
When you save the same object in a pickle it is written once, and for the 
second occurence a reference to the previously stored object is written:

>>> import cPickle
>>> import pickletools
>>> data = ("alpha", "alpha")
>>> cPickle.dumps(data)
"(S'alpha'\np1\ng1\ntp2\n."
>>> dump = _
>>> pickletools.dis(dump)
    0: (    MARK
    1: S        STRING     'alpha'
   10: p        PUT        1
   13: g        GET        1
   16: t        TUPLE      (MARK at 0)
   17: p    PUT        2
   20: .    STOP
highest protocol among opcodes = 0

Now let's break it:

>>> broken = dump.replace("g1", "g2")
>>> cPickle.loads(broken)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cPickle.BadPickleGet: 2

> I assume the data file is corrupt but the question is can I fix it with
> an editor using that info?
 
Probably not. You could run your file through pickletools.dis() to find the 
location of the bad key

>>> pickletools.dis(broken)
    0: (    MARK
    1: S        STRING     'alpha'
   10: p        PUT        1
   13: g        GET        2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/pickletools.py", line 2006, in dis
    raise ValueError(errormsg)
ValueError: memo key 2 has never been stored into

but if you replace it with a valid one your data will still be messed up. 
Also, there are probably other errors in the file.

Another idea, patching the Unpickler implemented in Python to replace bad 
references with a dummy object:

>>> import pickle
>>> class U(pickle.Unpickler):
...     def my_load_get(self):
...             try:
...                     obj = self.load_get()
...             except KeyError:
...                     obj = "<object could not be retrieved>"
...             self.append(obj)
...     dispatch = pickle.Unpickler.dispatch.copy()
...     dispatch[pickle.GET] = my_load_get
...
>>> from cStringIO import StringIO
>>> U(StringIO(broken)).load()
('alpha', '<object could not be retrieved>')

suffers from the same problems.