Unpickling data with classes from dynamic modules

[Prasad, Ramit]

Fredrik Tolf wrote:
> 
> Dear list,
> 
> I have a system in which I load modules dynamically every now and then
> (that is, creating a module with types.ModuleType, compiling the code for
> the module and then executing it in the module with exec()), and where I
> would wish to be able to have classes in those modules containing classes
> that could pickled, and then unpickled again.
> 
> The problem with that, currently, is of course two-fold: The dynamically
> loaded module may not be loaded at the time when the unpickling takes
> place, but even if it were, it isn't registered in sys.modules, so the
> unpickler can't find it either way. And, of course, that is detected
> already when pickling.
> 
> Is there any way to fix this, at all?
> 
> I considered trying to create subclasses of the pickler and unpickler that
> pickle a reference to a module loader and data for the particular module
> along with a class that comes from such a module, by overriding
> Pickler.save_global and Unpickler.load_global. Unfortunately, however,
> those functions aren't part of the public interface and can't be
> overridden when the C-pickle module is used instead (which, obviously, is
> what normally happens).
> 
> Is there any way around this without having to modify the pickle module
> itself?
> 
> --
> 
> Fredrik Tolf
> --

I believe rather than subclassing the pickler, you are expected to 
change the behavior from within the class via __getstate__ and __setstate__.

http://docs.python.org/2/library/pickle.html#object.__getstate__

Although, for your use case (loading unknown classes) the section on pickle and 
extension types may be more appropriate.

http://docs.python.org/2/library/pickle.html#pickling-and-unpickling-extension-types

Maybe all you need to add is implementation for obj.__reduce__


~Ramit



This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.