[Python-Dev] Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues

[Dimitrios Pritsos]

On 12/16/2010 02:29 PM, Michael Foord wrote:
> On 16/12/2010 11:09, Dimitrios Pritsos wrote:
>>
>> Hello Core Developers,
>>
>> My name is Dimitrios and I am newbie in python. I am working on a 
>> Project (part of my PhD) that is called Synergeticprocessing module. 
>> Initially is imitating the multiprocessing built in module but the 
>> processes are distributed on a LAN and not Locally. The main issue I 
>> have is with Pickle module. And I think I found some kind of BUG in 
>> the built in multiprocessing module.
>>
>
> Hello Dimitrios,
>
> Please post your bug report to the Python bug tracker. As you think 
> you have a fix for the issue it is much more likely to be applied 
> quickly if you can package it in the form of a test that demonstrates 
> the issue and a patch that fixes it.
>
> http://bugs.python.org/
>
> All the best,
>
> Michael Foord
>



Hello Michael,

OK I will do sent it to the bug tracker. But what about the last issue 
i.e. that even if the class is transfered-and-pickled-unpickled it 
raises an exception if the class is defined into the __main__ script and 
not imported from an other one. (see below) could you please guide me 
where should I post this for someone to help me fix it!

Best Regards,

Dimitrios




>
>
>> (Synergeticprocessing module is located at GitHub: 
>> https://github.com/dpritsos/synergeticprocessing)
>>
>> Starting with the "BUG". In case someone uses the 
>> multiprocessing.Pool of processes he/she has to face the problem of 
>> types.MehtodType Impossible pickling. That is you cannot dispatch an 
>> class instance method to the to the Process Pool. However, digging in 
>> to the Source Code of the module there are few lines that resolve 
>> this issue however are not activated or they are faultily activated 
>> so they do not work. This is the 'BUG'
>>
>> _@ ....../multiprocessing/forking.py_
>>
>>
>> .
>> .
>> .
>>
>> #
>> # Try making some callable types picklable
>> #
>>
>> from pickle import Pickler
>> class ForkingPickler(Pickler):
>>     dispatch = Pickler.dispatch.copy()
>>
>>     @classmethod
>>     def register(cls, type, reduce):
>>         def dispatcher(self, obj):
>>             rv = reduce(obj)
>>             self.save_reduce(obj=obj, *rv)
>>         cls.dispatch[type] = dispatcher
>>
>> def _reduce_method(m):
>>     if m.im_self is None:
>>         return getattr, (m.im_class, m.im_func.func_name)
>>     else:
>>         return getattr, (m.im_self, m.im_func.func_name)
>> ForkingPickler.register(type(ForkingPickler.save), _reduce_method)
>>
>> def _reduce_method_descriptor(m):
>>     return getattr, (m.__objclass__, m.__name__)
>> ForkingPickler.register(type(list.append), _reduce_method_descriptor)
>> ForkingPickler.register(type(int.__add__), _reduce_method_descriptor)
>>
>> #def _reduce_builtin_function_or_method(m):
>> #    return getattr, (m.__self__, m.__name__)
>> #ForkingPickler.register(type(list().append), 
>> _reduce_builtin_function_or_method)
>> #ForkingPickler.register(type(int().__add__), 
>> _reduce_builtin_function_or_method)
>> .
>> .
>> .
>>
>> The RED lines are not doing the job, for some reason they are not 
>> managing to register the GREEN function as a global reduce/pickling 
>> function even if you call the registration function into you __main__ 
>> script.
>>
>> The solution I found is just to do this
>> *
>> import copy_reg
>> import types*
>>
>> def _reduce_method(m):
>>     if m.im_self is None:
>>         return getattr, (m.im_class, m.im_func.func_name)
>>     else:
>>         return getattr, (m.im_self, m.im_func.func_name)
>>
>> *copy_reg.pickle(types.MethodType, _reduce_method)*
>> .
>> .
>> .
>>
>> Doing that everything works FINE. But ONLY for local methods i.e. the 
>> ones that their class is defined on the __main__ script or other 
>> import-ed.
>>
>> In case you want to send something remotely (in an other machine) or 
>> to an other __main__ script running separately then you get a message 
>> like this:
>>
>> 'module' object has no attribute '<my_class>'
>>
>> The only way to resolve this is firstly to import a script that has 
>> <my_class> defined there and everything works fine.
>>
>> SO the problems it seems to be that the *m.im_class*  (look code 
>> above) has some attribute __module__ defined as __module__ = 
>> '__main__' or something like that. And this is the reason why remote 
>> script cannot execute the function. I mean that the _reduce_method() 
>> above DOES is pickling the whole CLASS object so there is no reason 
>> not to be executed at the remote script. Besides it does as mentioned 
>> above in you just import this the user defined class form an other 
>> script.
>>
>>
>> I have already spent about 12 weeks working on building my 
>> synergeticPool and resolve the issue of Pickling and only 2 days 
>> needed for the code of the Pool and the rest of the time was spent 
>> for the Pickling issues, and study all the Class related mechanics of 
>> python. That was the reason I ve started digging the 
>> multipocessessing module and found this say 'BUG', and finally sent 
>> this email.
>>
>> Best Regards,
>>
>>
>> Dimitrios
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>
>
> -- 
>
> http://www.voidspace.org.uk/
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
> May you share freely, never taking more than you give.
> -- the sqlite blessinghttp://www.sqlite.org/different.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20101216/b5fc7d38/attachment.html>