multiprocessing question/error

[Eduardo Lenz]

On Saturday 17 January 2009 00:43:35 Aaron Brady wrote:
> On Jan 16, 11:39 pm, Eduardo Lenz <l... at joinville.udesc.br> wrote:
> > Hi,
> >
> > I was using the former processing package with python 2.5 with no
> > problems. After switching to python 2.6.1 I am having some problems with
> > the same code. The problem seems to be related to the fact that I am
> > using Pool.map with a bounded method, since it is inside a class. To
> > clarify a little bit, let me show some parts of the code ....
>
> Hello,
>
> > class Pygen3(self)....
>
> I infer this was a typo.
>
> >     ....
> >     ....
> >     ....
> >     def calcula(self,indiv):
> >         ....
> >         ....
> >         ....
> >
> >    def evaluate(self):
> >      ....
> >      ....
> >     indiv = range(mult*self.popsize,(mult+1)*self.popsize)
> >    pool = Pool(processes=nproc)
> >    results = pool.map(self.calcula,indiv)
>
> I infer the indentation drifted.
>
> >    ...
> >    ...
> >
> > the error is the following
> >
> > Exception in thread Thread-1:
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
> >     self.run()
> >   File "/usr/lib/python2.6/threading.py", line 477, in run
> >     self.__target(*self.__args, **self.__kwargs)
> >   File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> > _handle_tasks
> >     put(task)
> > PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
> > __builtin__.instancemethod failed
> >
> > Thanks for your help.
>
> Checking on two typos above.  I have this working:
>
> from multiprocessing import Process
>
> class C(object):
>     def m1( self ):
>         p1 = Process(target=self.m2, args=('bob1',))
>         p2 = Process(target=self.m2, args=('bob2',))
>         p1.start()
>         p2.start()
>         p1.join()
>         p2.join()
>     def m2( self, name ):
>         print 'hello', name
>
> if __name__ == '__main__':
>     c= C()
>     c.m1()
>
> But I get the same error as you with this:
>
> from multiprocessing import Process, Pool
>
> class C(object):
>     def m1( self ):
>         arg= range( 10 )
>         pool= Pool( 5 )
>         results = pool.map(self.m2,arg)
>         print results
>     def m2( self, name ):
>         print 'hello', name
>
> if __name__ == '__main__':
>     c= C()
>     c.m1()
>
> I build a workaround with this:
>
> from multiprocessing import Process, Pool
>
> class C(object):
>     def m1( self ):
>         arg= zip( [ self ]* 10, range( 10 ) )
>         pool= Pool( 5 )
>         results = pool.map(C_m2,arg)
>         print results
>     def m2( self, arg ):
>         print 'hello', arg
>         return arg* 2
> def C_m2( ar, **kwar ):
>     return C.m2( *ar, **kwar )
>
> if __name__ == '__main__':
>     c= C()
>     c.m1()
>
> Note that you have to explicitly include 'self' as a first parameter
> when packing the arguments, and then explicitly unpack it in the
> helper function.  Also, wrapping 'm2' with '@staticmethod' also fails
> with this error:
>
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "c:\programs\python26\lib\threading.py", line 522, in
> __bootstrap_inner
>     self.run()
>   File "c:\programs\python26\lib\threading.py", line 477, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "c:\programs\python26\lib\multiprocessing\pool.py", line 225,
> in _handle_tasks
>     put(task)
> PicklingError: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed
>
> If this works for you, it's possible to make the helper function more
> dynamic, by accepting a class name and an attribute name as string
> arguments, then retrieving them in the subprocesses.
> --
> http://mail.python.org/mailman/listinfo/python-list

Thanks for your help ! The only question remaninig is why it iused to work 
with the former processing package. Another question: is it possible to 
change a bound method to a function on the fly ? I was wondering if something 
like class.__dict__["function_name"] could do the job. Actually, I was 
wondering with something like a casting, but a guess it is not possible.

Thanks again,
Eduardo.
-- 

 Eduardo Lenz Cardoso
 Dr.  Eng.
 Associate Professor
 
 State University of Santa Catarina
 Department of Mechanical Engineering
 89223-100 - Joinville-SC - Brasil

 Tel: +55 47 4009-7971 - Fax: +55 47 4009-7940
 E-mail: lenz at Joinville.udesc.br          
 ---------------------------------------------

-- 
Esta mensagem foi verificada pelo sistema de antivírus e
 acredita-se estar livre de perigo.