writing pickle function

[Steve Holden]

Peter Otten wrote:
> Steve Holden wrote:
> 
>> Peter Otten wrote:
>>> perfreem at gmail.com wrote:
>>>
>>>> i am using nested defaultdict from collections and i would like to
>>>> write it as a pickle object to a file. when i try:
>>>>
>>>> from collections import defaultdict
>>>> x = defaultdict(lambda: defaultdict(list))
>>>>
>>>> and then try to write to a pickle file, it says:
>>>>
>>>> TypeError: can't pickle function objects
>>>>
>>>> is there a way around this? it's simply a dictionary that i want to
>>>> write to file.. this works no problems with ordinary dicts.
>>> The error message is misleading. You can pickle your defaultdict if you
>>> use a "normal" function instead of the lambda:
>>>
>>> $ cat pickle_defaultdict.py
>>> import sys
>>> from collections import defaultdict
>>> from cPickle import dumps, loads
>>>
>>> if "--lambda" in sys.argv:
>>>     make_inner = lambda: defaultdict(list)
>>> else:
>>>     def make_inner():
>>>         return defaultdict(list)
>>>
>>> d = defaultdict(make_inner)
>>> d[1][2].append(42)
>>> e = loads(dumps(d))
>>> print e == d
>>>
>>> $ python pickle_defaultdict.py
>>> True
>>> $ python pickle_defaultdict.py --lambda
>>> Traceback (most recent call last):
>>>   File "pickle_defaultdict.py", line 13, in <module>
>>>     e = loads(dumps(d))
>>>   File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
>>>     raise TypeError, "can't pickle %s objects" % base.__name__
>>> TypeError: can't pickle function objects
>>>
>> But can you unpickle the objects you pickle? I believe that functions
>> are references relative to the modules they come from, and so won't be
>> found unless the same module can be imported at unpickle time. Could be
>> wrong, haven't read the source, but I understand that functions work the
>> same as classes.
> 
> Indeed, what is stored in the pickle is the module and function name, not
> the byte code:
> 
>>>> from cPickle import dumps, loads
>>>> def f(): print "original"
> ...
>>>> s = dumps(f)
>>>> def f(): print "updated"
> ...
>>>> g = loads(s)
>>>> g()
> updated
> 
> That limits pickle as an exchange format to installations that provide
> compatible versions of the pickled classes and functions. 
> 
> Do you think that would be a problem for the OP?
> 
Nice demonstration!

Possibly not, though the original use of lambdas demonstrated at least
confusion. But there needs to be an understanding that the pickled
function has to be importable. Just using a function in __main__ and
then trying to unpickle from another program that doesn't contain the
function won;t hack it.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/