[execnet-dev] help

[Samy Vilar]

I'll try, but do you know of a way to use execnet like pool =
multiprocessing.Pool(processes = 4); results =  pool.mapt(func, values)

On Sat, Mar 5, 2011 at 4:34 PM, holger krekel <holger at merlinux.eu> wrote:

> On Sat, Mar 05, 2011 at 15:55 -0500, Samy Vilar wrote:
> > yep ...
>
> Without checking deeper I don't know what is going on exactly. I guess
> multiprocessing uses pickling and that doesn't seem to work with the way
> the execnet subprocess is setup.  Have you tried just using execnet to
> distribute your computation tasks?
>
> some examples from this page might help you if you try that:
>
>    http://codespeak.net/execnet/examples.html
>
> best,
> holger
>
> P.S.: please keep execnet-dev in CC because otherwise the
> other subscribers cannot chime in.
>
> > On Sat, Mar 5, 2011 at 3:48 PM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > >
> > >
> > > On Sat, Mar 05, 2011 at 14:32 -0500, Samy Vilar wrote:
> > > > >>> gw =
> > > >
> > >
> execnet.makegateway('ssh=flemming//python=/home1/student/svilar/Development/machines/flemming/python/python')
> > > > >>> ch = gw.remote_exec("""
> > > > ... import multiprocessing
> > > > ... def f(h):
> > > > ...     print h
> > > > ... a = [1,2,3,4]
> > > > ... pool = multiprocessing.Pool(processes = 4)
> > > > ... pool.map(f, a)
> > > > ... pool.close()
> > > > ... pool.join()
> > > > ... channel.send('done')
> > > > ... """)
> > > > >>> Exception in thread Thread-2:
> > >
> > > Does this work if you use it in an interactive PYthon prompt
> > > without any execnet?
> > >
> > > holger
> > >
> > >
> > > > Traceback (most recent call last):
> > > >   File
> > > >
> > >
> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> > > > line 530, in __bootstrap_inner
> > > >     self.run()
> > > >   File
> > > >
> > >
> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> > > > line 483, in run
> > > >     self.__target(*self.__args, **self.__kwargs)
> > > >   File
> > > >
> > >
> "/home1/student/svilar/Development/machines/flemming/python/Lib/multiprocessing/pool.py",
> > > > line 285, in _handle_tasks
> > > >     put(task)
> > > > PicklingError: Can't pickle <type 'function'>: attribute lookup
> > > > __builtin__.function failed
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Mar 5, 2011 at 9:11 AM, holger krekel <holger at merlinux.eu>
> > > wrote:
> > > >
> > > > > On Sat, Mar 05, 2011 at 02:16 -0500, Samy Vilar wrote:
> > > > > > Is it possible to open a multiprocessing pool remotely though
> > > execnet? I
> > > > > > keep getting "PicklingError: Can't pickle <type 'function'>:
> > > attribute
> > > > > > lookup __builtin__.function failed" running on python 2.7 with
> > > execnet
> > > > > 1.0.9
> > > > >
> > > > > Can you send the code fragment you try to execute and that doesn't
> > > > > work?
> > > > >
> > > > > FWIW you can not pass code or class instances across an execnet
> > > channel.
> > > > >
> > > > > best,
> > > > > holger
> > > > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110305/ce1a332a/attachment.html>