[Python-Dev] Pickling problems are hard to debug

[Gary Poster]

On Mar 25, 2006, at 8:13 PM, Greg Ewing wrote:

> There seems to be a need for better diagnostics
> when pickle encounters something that can't be
> pickled.
>
> Recently when attempting to pickle a rather
> large and complicated data structure, I got
> the following incomprehensible message:
>
>    cPickle.PicklingError: args[0]
>      from __newobj__ args has the wrong class
>
> Trying again with protocol 1 instead of 2,
> I get
>
>    TypeError: can't pickle function objects
>
> which I'm *guessing* is because somewhere I've
> tried to pickle a nested function or a bound
> method. But it still doesn't give me any idea
> *which* function I tried to pickle or where
> abouts it turns up in the data structure.
>
> Anyone have any ideas how the situation could
> be improved? At the very least, it could
> include some info about the type and identity
> of the offending object.

You are asking for ideas on how to change the pickle story to help.   
However, just reading your issue, I thought I might have done a  
debugging hack like this, at least for the protocol 1 traceback.

We'll assume that the error is more mysterious than what I've  
manufactured here.

 >>> import cPickle
 >>> cPickle.dumps({'foo': lambda: 42})
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
python2.4/copy_reg.py", line 69, in _reduce_ex
     raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects
 >>> import copy_reg
 >>> def debug(obj):
...     import pdb; pdb.set_trace()
...
 >>> import types
 >>> copy_reg.pickle(types.FunctionType, debug)
 >>> cPickle.dumps({'foo': lambda: 42})
--Return--
 > <stdin>(2)debug()->None
(Pdb) p obj
<function <lambda> at 0x63230>
(Pdb) p obj.__module__
'__main__'

I also might have used pickle, rather than cPickle, to try and see  
what happened, if that ended up being necessary.

I don't use protocol 2 much: that error message in particular looked  
a bit difficult, and my hack might not be any help there.  I agree  
that it would be nice to have a better message there, in particular.

back to lurking...

Gary