[Python-Dev] Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues

[Michael Foord]

On 16/12/2010 11:09, Dimitrios Pritsos wrote:
>
> Hello Core Developers,
>
> My name is Dimitrios and I am newbie in python. I am working on a 
> Project (part of my PhD) that is called Synergeticprocessing module. 
> Initially is imitating the multiprocessing built in module but the 
> processes are distributed on a LAN and not Locally. The main issue I 
> have is with Pickle module. And I think I found some kind of BUG in 
> the built in multiprocessing module.
>

Hello Dimitrios,

Please post your bug report to the Python bug tracker. As you think you 
have a fix for the issue it is much more likely to be applied quickly if 
you can package it in the form of a test that demonstrates the issue and 
a patch that fixes it.

http://bugs.python.org/

All the best,

Michael Foord



> (Synergeticprocessing module is located at GitHub: 
> https://github.com/dpritsos/synergeticprocessing)
>
> Starting with the "BUG". In case someone uses the multiprocessing.Pool 
> of processes he/she has to face the problem of types.MehtodType 
> Impossible pickling. That is you cannot dispatch an class instance 
> method to the to the Process Pool. However, digging in to the Source 
> Code of the module there are few lines that resolve this issue however 
> are not activated or they are faultily activated so they do not work. 
> This is the 'BUG'
>
> _@ ....../multiprocessing/forking.py_
>
>
> .
> .
> .
>
> #
> # Try making some callable types picklable
> #
>
> from pickle import Pickler
> class ForkingPickler(Pickler):
>     dispatch = Pickler.dispatch.copy()
>
>     @classmethod
>     def register(cls, type, reduce):
>         def dispatcher(self, obj):
>             rv = reduce(obj)
>             self.save_reduce(obj=obj, *rv)
>         cls.dispatch[type] = dispatcher
>
> def _reduce_method(m):
>     if m.im_self is None:
>         return getattr, (m.im_class, m.im_func.func_name)
>     else:
>         return getattr, (m.im_self, m.im_func.func_name)
> ForkingPickler.register(type(ForkingPickler.save), _reduce_method)
>
> def _reduce_method_descriptor(m):
>     return getattr, (m.__objclass__, m.__name__)
> ForkingPickler.register(type(list.append), _reduce_method_descriptor)
> ForkingPickler.register(type(int.__add__), _reduce_method_descriptor)
>
> #def _reduce_builtin_function_or_method(m):
> #    return getattr, (m.__self__, m.__name__)
> #ForkingPickler.register(type(list().append), 
> _reduce_builtin_function_or_method)
> #ForkingPickler.register(type(int().__add__), 
> _reduce_builtin_function_or_method)
> .
> .
> .
>
> The RED lines are not doing the job, for some reason they are not 
> managing to register the GREEN function as a global reduce/pickling 
> function even if you call the registration function into you __main__ 
> script.
>
> The solution I found is just to do this
> *
> import copy_reg
> import types*
>
> def _reduce_method(m):
>     if m.im_self is None:
>         return getattr, (m.im_class, m.im_func.func_name)
>     else:
>         return getattr, (m.im_self, m.im_func.func_name)
>
> *copy_reg.pickle(types.MethodType, _reduce_method)*
> .
> .
> .
>
> Doing that everything works FINE. But ONLY for local methods i.e. the 
> ones that their class is defined on the __main__ script or other 
> import-ed.
>
> In case you want to send something remotely (in an other machine) or 
> to an other __main__ script running separately then you get a message 
> like this:
>
> 'module' object has no attribute '<my_class>'
>
> The only way to resolve this is firstly to import a script that has 
> <my_class> defined there and everything works fine.
>
> SO the problems it seems to be that the *m.im_class*  (look code 
> above) has some attribute __module__ defined as __module__ = 
> '__main__' or something like that. And this is the reason why remote 
> script cannot execute the function. I mean that the _reduce_method() 
> above DOES is pickling the whole CLASS object so there is no reason 
> not to be executed at the remote script. Besides it does as mentioned 
> above in you just import this the user defined class form an other 
> script.
>
>
> I have already spent about 12 weeks working on building my 
> synergeticPool and resolve the issue of Pickling and only 2 days 
> needed for the code of the Pool and the rest of the time was spent for 
> the Pickling issues, and study all the Class related mechanics of 
> python. That was the reason I ve started digging the multipocessessing 
> module and found this say 'BUG', and finally sent this email.
>
> Best Regards,
>
>
> Dimitrios
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20101216/7d6691f3/attachment.html>