[execnet-dev] help

[holger krekel]

On Sun, Mar 06, 2011 at 19:09 -0500, Samy Vilar wrote:
> I keep getting "can't serialize <type 'numpy.ndarray'>" numpy arrays can be
> pickled but for something reason execnet can't seem to serialize it ... and
> converting to a list kills performance both clock cycles and memory

to be clear: execnet does not use pickle for sending data through channels.
It only supports serialization of basic types like lists, dicts, ints,
strings, etc.  You can try, however, to use pickle.dumps/pickle.loads
and send the resulting string over a channel.

holger

> On Sat, Mar 5, 2011 at 9:07 PM, Samy Vilar <samy.vilar at gmail.com> wrote:
> 
> > I'll try, but do you know of a way to use execnet like pool =
> > multiprocessing.Pool(processes = 4); results =  pool.mapt(func, values)
> >
> >
> > On Sat, Mar 5, 2011 at 4:34 PM, holger krekel <holger at merlinux.eu> wrote:
> >
> >> On Sat, Mar 05, 2011 at 15:55 -0500, Samy Vilar wrote:
> >> > yep ...
> >>
> >> Without checking deeper I don't know what is going on exactly. I guess
> >> multiprocessing uses pickling and that doesn't seem to work with the way
> >> the execnet subprocess is setup.  Have you tried just using execnet to
> >> distribute your computation tasks?
> >>
> >> some examples from this page might help you if you try that:
> >>
> >>    http://codespeak.net/execnet/examples.html
> >>
> >> best,
> >> holger
> >>
> >> P.S.: please keep execnet-dev in CC because otherwise the
> >> other subscribers cannot chime in.
> >>
> >> > On Sat, Mar 5, 2011 at 3:48 PM, holger krekel <holger at merlinux.eu>
> >> wrote:
> >> >
> >> > >
> >> > >
> >> > > On Sat, Mar 05, 2011 at 14:32 -0500, Samy Vilar wrote:
> >> > > > >>> gw =
> >> > > >
> >> > >
> >> execnet.makegateway('ssh=flemming//python=/home1/student/svilar/Development/machines/flemming/python/python')
> >> > > > >>> ch = gw.remote_exec("""
> >> > > > ... import multiprocessing
> >> > > > ... def f(h):
> >> > > > ...     print h
> >> > > > ... a = [1,2,3,4]
> >> > > > ... pool = multiprocessing.Pool(processes = 4)
> >> > > > ... pool.map(f, a)
> >> > > > ... pool.close()
> >> > > > ... pool.join()
> >> > > > ... channel.send('done')
> >> > > > ... """)
> >> > > > >>> Exception in thread Thread-2:
> >> > >
> >> > > Does this work if you use it in an interactive PYthon prompt
> >> > > without any execnet?
> >> > >
> >> > > holger
> >> > >
> >> > >
> >> > > > Traceback (most recent call last):
> >> > > >   File
> >> > > >
> >> > >
> >> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> >> > > > line 530, in __bootstrap_inner
> >> > > >     self.run()
> >> > > >   File
> >> > > >
> >> > >
> >> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> >> > > > line 483, in run
> >> > > >     self.__target(*self.__args, **self.__kwargs)
> >> > > >   File
> >> > > >
> >> > >
> >> "/home1/student/svilar/Development/machines/flemming/python/Lib/multiprocessing/pool.py",
> >> > > > line 285, in _handle_tasks
> >> > > >     put(task)
> >> > > > PicklingError: Can't pickle <type 'function'>: attribute lookup
> >> > > > __builtin__.function failed
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sat, Mar 5, 2011 at 9:11 AM, holger krekel <holger at merlinux.eu>
> >> > > wrote:
> >> > > >
> >> > > > > On Sat, Mar 05, 2011 at 02:16 -0500, Samy Vilar wrote:
> >> > > > > > Is it possible to open a multiprocessing pool remotely though
> >> > > execnet? I
> >> > > > > > keep getting "PicklingError: Can't pickle <type 'function'>:
> >> > > attribute
> >> > > > > > lookup __builtin__.function failed" running on python 2.7 with
> >> > > execnet
> >> > > > > 1.0.9
> >> > > > >
> >> > > > > Can you send the code fragment you try to execute and that doesn't
> >> > > > > work?
> >> > > > >
> >> > > > > FWIW you can not pass code or class instances across an execnet
> >> > > channel.
> >> > > > >
> >> > > > > best,
> >> > > > > holger
> >> > > > >
> >> > >
> >>
> >
> >