pickling instances of metaclass generated classes

[lars van gemerden]

On Dec 29, 8:55 pm, Ian Kelly <ian.g.ke... at gmail.com> wrote:
> On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <l... at rational-it.com> wrote:
>
> > Hello,
>
> > Can someone help me with the following:
>
> > I am using metaclasses to make classes and these classes to make
> > instances. Now I want to use multiprocessing, which needs to pickle
> > these instances.
>
> > Pickle cannot find the class definitions of the instances. I am trying
> > to add a line to the __new__ of the metaclass to add the new class
> > under the right name in the right module/place, so pickle can find
> > it.
>
> > Is this the right approach? Can anyone explain to me where/how to add
> > these classes for pickle to find and maybe why?
>
> It sounds like you're trying to do something like this?
>
> >>> class MetaClass(type):
>
> ...     pass
> ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
> >>> instance
>
> <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
> >>> pickle.dumps(instance)
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "c:\python27\lib\pickle.py", line 1374, in dumps
>     Pickler(file, protocol).dump(obj)
>   File "c:\python27\lib\pickle.py", line 224, in dump
>     self.save(obj)
>   File "c:\python27\lib\pickle.py", line 331, in save
>     self.save_reduce(obj=obj, *rv)
>   File "c:\python27\lib\pickle.py", line 401, in save_reduce
>     save(args)
>   File "c:\python27\lib\pickle.py", line 286, in save
>     f(self, obj) # Call unbound method with explicit self
>   File "c:\python27\lib\pickle.py", line 562, in save_tuple
>     save(element)
>   File "c:\python27\lib\pickle.py", line 295, in save
>     self.save_global(obj)
>   File "c:\python27\lib\pickle.py", line 748, in save_global
>     (obj, module, name))
> pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
> it's not found as __main__.<Anonymous>
>
> Yeah, pickle's not going to work with anonymous classes.  As you
> suggest, you could dynamically add the classes to the module namespace
> so that pickle.dumps will find them, but bear in mind that they will
> also have to exist when calling pickle.loads, so you will need to be
> able to reconstruct the same anonymous classes before unpickling later
> on.
>
> Cheers,
> Ian

Ian also wrote:

'''
Actually, I was wrong, you probably don't need to do that.  I suggest
going with Robert Kern's suggestion to either register the class with
the copy_reg module, or (perhaps better since it won't leak
registrations) implement a __reduce__ method on the class.  For
example, this seems to work:

>>> def reconstructor(*metaclass_args):
...     cls = MetaClass.build_class(*metaclass_args)
...     self = cls.__new__(cls)
...     return self
...
>>> class MetaClass(type):
...     @classmethod
...     def build_class(mcs, arg1, arg2, arg3):
...         # Do something useful with the args...
...         class _AnonymousClass(object):
...             __metaclass__ = mcs
...             def __reduce__(self):
...                 return (reconstructor, ('foo', 'bar', 'baz'),
self.__dict__)
...         return _AnonymousClass
...
>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
>>> instance
<__main__._AnonymousClass object at 0x011DB410>
>>> instance.banana = 42
>>> import pickle
>>> s = pickle.dumps(instance)
>>> s
"c__main__\nreconstructor
\np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI42\nsb."
>>> inst2 = pickle.loads(s)
>>> inst2
<__main__._AnonymousClass object at 0x011DBE90>
>>> inst2.banana
42
>>> inst2.__class__ is instance.__class__
False

Cheers,
Ian

'''