From samy.vilar at gmail.com  Sat Mar  5 08:16:08 2011
From: samy.vilar at gmail.com (Samy Vilar)
Date: Sat, 5 Mar 2011 02:16:08 -0500
Subject: [execnet-dev] help
Message-ID: <AANLkTi=AY_1SM_tZ0XZ1TyWdJZK0Db1Jk3NQkrLLufhD@mail.gmail.com>

Is it possible to open a multiprocessing pool remotely though execnet? I
keep getting "PicklingError: Can't pickle <type 'function'>: attribute
lookup __builtin__.function failed" running on python 2.7 with execnet 1.0.9

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110305/cb860657/attachment.html>

From holger at merlinux.eu  Sat Mar  5 15:09:23 2011
From: holger at merlinux.eu (holger krekel)
Date: Sat, 5 Mar 2011 14:09:23 +0000
Subject: [execnet-dev] help
In-Reply-To: <AANLkTikN5r=_bT=NrOCaTypd_Hn2d7U07mBrj_BTt9t2@mail.gmail.com>
References: <AANLkTikN5r=_bT=NrOCaTypd_Hn2d7U07mBrj_BTt9t2@mail.gmail.com>
Message-ID: <20110305140923.GN16231@merlinux.eu>

Hi Samy,

(the mail was queued by mailman, only arrived today, 5th March)

On Fri, Feb 25, 2011 at 18:31 -0500, Samy Vilar wrote:
> This has being driving me nuts for a while, I have a small network about
> half a dozen machines, all with python and third party modules installed
> in custom locations, now loading the appropriate python interpreter at each
> machines was trivial but every time I try to import a module I keep getting
> an error 'No module named ...', Im assuming, since execnet ssh with notty
> into a machine, bashrc or bashprofile isn't loaded and so the paths aren't
> being set, I've also tried adding the paths by remote_exec('import sys;
> sys.path.append(...)') and still nothing, I've even tried using chdir to
> point to my site-packages dir and still nothing, please help me? I'm
> currently using execnet version 1.0.9

Could you provide a small example of what doesn't work?

I'd expect "remote_exec('import sys ; sys.path.append(...)')"
to allow have imports afterwards succeeed.,

holger


From holger at merlinux.eu  Sat Mar  5 15:11:02 2011
From: holger at merlinux.eu (holger krekel)
Date: Sat, 5 Mar 2011 14:11:02 +0000
Subject: [execnet-dev] help
In-Reply-To: <AANLkTi=AY_1SM_tZ0XZ1TyWdJZK0Db1Jk3NQkrLLufhD@mail.gmail.com>
References: <AANLkTi=AY_1SM_tZ0XZ1TyWdJZK0Db1Jk3NQkrLLufhD@mail.gmail.com>
Message-ID: <20110305141102.GO16231@merlinux.eu>

On Sat, Mar 05, 2011 at 02:16 -0500, Samy Vilar wrote:
> Is it possible to open a multiprocessing pool remotely though execnet? I
> keep getting "PicklingError: Can't pickle <type 'function'>: attribute
> lookup __builtin__.function failed" running on python 2.7 with execnet 1.0.9

Can you send the code fragment you try to execute and that doesn't
work? 

FWIW you can not pass code or class instances across an execnet channel.

best,
holger


From samy.vilar at gmail.com  Sat Mar  5 20:22:03 2011
From: samy.vilar at gmail.com (Samy Vilar)
Date: Sat, 5 Mar 2011 14:22:03 -0500
Subject: [execnet-dev] help
In-Reply-To: <20110305140923.GN16231@merlinux.eu>
References: <AANLkTikN5r=_bT=NrOCaTypd_Hn2d7U07mBrj_BTt9t2@mail.gmail.com>
	<20110305140923.GN16231@merlinux.eu>
Message-ID: <AANLkTikFJRnA3a0xfCx_+dicx5qgWPzv=+VUrK3VWPR2@mail.gmail.com>

I fixed the issue by commenting out a line in my bashrc file for interactive
sessions, so its working now ...

On Sat, Mar 5, 2011 at 9:09 AM, holger krekel <holger at merlinux.eu> wrote:

> Hi Samy,
>
> (the mail was queued by mailman, only arrived today, 5th March)
>
> On Fri, Feb 25, 2011 at 18:31 -0500, Samy Vilar wrote:
> > This has being driving me nuts for a while, I have a small network about
> > half a dozen machines, all with python and third party modules installed
> > in custom locations, now loading the appropriate python interpreter at
> each
> > machines was trivial but every time I try to import a module I keep
> getting
> > an error 'No module named ...', Im assuming, since execnet ssh with notty
> > into a machine, bashrc or bashprofile isn't loaded and so the paths
> aren't
> > being set, I've also tried adding the paths by remote_exec('import sys;
> > sys.path.append(...)') and still nothing, I've even tried using chdir to
> > point to my site-packages dir and still nothing, please help me? I'm
> > currently using execnet version 1.0.9
>
> Could you provide a small example of what doesn't work?
>
> I'd expect "remote_exec('import sys ; sys.path.append(...)')"
> to allow have imports afterwards succeeed.,
>
> holger
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110305/40be06db/attachment.html>

From holger at merlinux.eu  Sat Mar  5 21:48:51 2011
From: holger at merlinux.eu (holger krekel)
Date: Sat, 5 Mar 2011 20:48:51 +0000
Subject: [execnet-dev] help
In-Reply-To: <AANLkTinf05A7AwkD4Ax4wU0YDAb9RmjpsNwGxKH-HtvH@mail.gmail.com>
References: <AANLkTi=AY_1SM_tZ0XZ1TyWdJZK0Db1Jk3NQkrLLufhD@mail.gmail.com>
	<20110305141102.GO16231@merlinux.eu>
	<AANLkTinf05A7AwkD4Ax4wU0YDAb9RmjpsNwGxKH-HtvH@mail.gmail.com>
Message-ID: <20110305204851.GQ16231@merlinux.eu>



On Sat, Mar 05, 2011 at 14:32 -0500, Samy Vilar wrote:
> >>> gw =
> execnet.makegateway('ssh=flemming//python=/home1/student/svilar/Development/machines/flemming/python/python')
> >>> ch = gw.remote_exec("""
> ... import multiprocessing
> ... def f(h):
> ...     print h
> ... a = [1,2,3,4]
> ... pool = multiprocessing.Pool(processes = 4)
> ... pool.map(f, a)
> ... pool.close()
> ... pool.join()
> ... channel.send('done')
> ... """)
> >>> Exception in thread Thread-2:

Does this work if you use it in an interactive PYthon prompt 
without any execnet?

holger


> Traceback (most recent call last):
>   File
> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> line 530, in __bootstrap_inner
>     self.run()
>   File
> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> line 483, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File
> "/home1/student/svilar/Development/machines/flemming/python/Lib/multiprocessing/pool.py",
> line 285, in _handle_tasks
>     put(task)
> PicklingError: Can't pickle <type 'function'>: attribute lookup
> __builtin__.function failed
> 
> 
> 
> 
> On Sat, Mar 5, 2011 at 9:11 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > On Sat, Mar 05, 2011 at 02:16 -0500, Samy Vilar wrote:
> > > Is it possible to open a multiprocessing pool remotely though execnet? I
> > > keep getting "PicklingError: Can't pickle <type 'function'>: attribute
> > > lookup __builtin__.function failed" running on python 2.7 with execnet
> > 1.0.9
> >
> > Can you send the code fragment you try to execute and that doesn't
> > work?
> >
> > FWIW you can not pass code or class instances across an execnet channel.
> >
> > best,
> > holger
> >


From holger at merlinux.eu  Sat Mar  5 22:34:27 2011
From: holger at merlinux.eu (holger krekel)
Date: Sat, 5 Mar 2011 21:34:27 +0000
Subject: [execnet-dev] help
In-Reply-To: <AANLkTiki-acLw0VnyMp2gvfNw6jgP5iY_x2uSNgS3pp4@mail.gmail.com>
References: <AANLkTi=AY_1SM_tZ0XZ1TyWdJZK0Db1Jk3NQkrLLufhD@mail.gmail.com>
	<20110305141102.GO16231@merlinux.eu>
	<AANLkTinf05A7AwkD4Ax4wU0YDAb9RmjpsNwGxKH-HtvH@mail.gmail.com>
	<20110305204851.GQ16231@merlinux.eu>
	<AANLkTiki-acLw0VnyMp2gvfNw6jgP5iY_x2uSNgS3pp4@mail.gmail.com>
Message-ID: <20110305213427.GR16231@merlinux.eu>

On Sat, Mar 05, 2011 at 15:55 -0500, Samy Vilar wrote:
> yep ...

Without checking deeper I don't know what is going on exactly. I guess
multiprocessing uses pickling and that doesn't seem to work with the way
the execnet subprocess is setup.  Have you tried just using execnet to
distribute your computation tasks?

some examples from this page might help you if you try that:

    http://codespeak.net/execnet/examples.html

best,
holger

P.S.: please keep execnet-dev in CC because otherwise the
other subscribers cannot chime in.

> On Sat, Mar 5, 2011 at 3:48 PM, holger krekel <holger at merlinux.eu> wrote:
> 
> >
> >
> > On Sat, Mar 05, 2011 at 14:32 -0500, Samy Vilar wrote:
> > > >>> gw =
> > >
> > execnet.makegateway('ssh=flemming//python=/home1/student/svilar/Development/machines/flemming/python/python')
> > > >>> ch = gw.remote_exec("""
> > > ... import multiprocessing
> > > ... def f(h):
> > > ...     print h
> > > ... a = [1,2,3,4]
> > > ... pool = multiprocessing.Pool(processes = 4)
> > > ... pool.map(f, a)
> > > ... pool.close()
> > > ... pool.join()
> > > ... channel.send('done')
> > > ... """)
> > > >>> Exception in thread Thread-2:
> >
> > Does this work if you use it in an interactive PYthon prompt
> > without any execnet?
> >
> > holger
> >
> >
> > > Traceback (most recent call last):
> > >   File
> > >
> > "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> > > line 530, in __bootstrap_inner
> > >     self.run()
> > >   File
> > >
> > "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> > > line 483, in run
> > >     self.__target(*self.__args, **self.__kwargs)
> > >   File
> > >
> > "/home1/student/svilar/Development/machines/flemming/python/Lib/multiprocessing/pool.py",
> > > line 285, in _handle_tasks
> > >     put(task)
> > > PicklingError: Can't pickle <type 'function'>: attribute lookup
> > > __builtin__.function failed
> > >
> > >
> > >
> > >
> > > On Sat, Mar 5, 2011 at 9:11 AM, holger krekel <holger at merlinux.eu>
> > wrote:
> > >
> > > > On Sat, Mar 05, 2011 at 02:16 -0500, Samy Vilar wrote:
> > > > > Is it possible to open a multiprocessing pool remotely though
> > execnet? I
> > > > > keep getting "PicklingError: Can't pickle <type 'function'>:
> > attribute
> > > > > lookup __builtin__.function failed" running on python 2.7 with
> > execnet
> > > > 1.0.9
> > > >
> > > > Can you send the code fragment you try to execute and that doesn't
> > > > work?
> > > >
> > > > FWIW you can not pass code or class instances across an execnet
> > channel.
> > > >
> > > > best,
> > > > holger
> > > >
> >


From samy.vilar at gmail.com  Sun Mar  6 03:07:36 2011
From: samy.vilar at gmail.com (Samy Vilar)
Date: Sat, 5 Mar 2011 21:07:36 -0500
Subject: [execnet-dev] help
In-Reply-To: <20110305213427.GR16231@merlinux.eu>
References: <AANLkTi=AY_1SM_tZ0XZ1TyWdJZK0Db1Jk3NQkrLLufhD@mail.gmail.com>
	<20110305141102.GO16231@merlinux.eu>
	<AANLkTinf05A7AwkD4Ax4wU0YDAb9RmjpsNwGxKH-HtvH@mail.gmail.com>
	<20110305204851.GQ16231@merlinux.eu>
	<AANLkTiki-acLw0VnyMp2gvfNw6jgP5iY_x2uSNgS3pp4@mail.gmail.com>
	<20110305213427.GR16231@merlinux.eu>
Message-ID: <AANLkTinYhA8FvtZA_f_rkzjtkXVR=OG4MVbdqVuNDeXi@mail.gmail.com>

I'll try, but do you know of a way to use execnet like pool =
multiprocessing.Pool(processes = 4); results =  pool.mapt(func, values)

On Sat, Mar 5, 2011 at 4:34 PM, holger krekel <holger at merlinux.eu> wrote:

> On Sat, Mar 05, 2011 at 15:55 -0500, Samy Vilar wrote:
> > yep ...
>
> Without checking deeper I don't know what is going on exactly. I guess
> multiprocessing uses pickling and that doesn't seem to work with the way
> the execnet subprocess is setup.  Have you tried just using execnet to
> distribute your computation tasks?
>
> some examples from this page might help you if you try that:
>
>    http://codespeak.net/execnet/examples.html
>
> best,
> holger
>
> P.S.: please keep execnet-dev in CC because otherwise the
> other subscribers cannot chime in.
>
> > On Sat, Mar 5, 2011 at 3:48 PM, holger krekel <holger at merlinux.eu>
> wrote:
> >
> > >
> > >
> > > On Sat, Mar 05, 2011 at 14:32 -0500, Samy Vilar wrote:
> > > > >>> gw =
> > > >
> > >
> execnet.makegateway('ssh=flemming//python=/home1/student/svilar/Development/machines/flemming/python/python')
> > > > >>> ch = gw.remote_exec("""
> > > > ... import multiprocessing
> > > > ... def f(h):
> > > > ...     print h
> > > > ... a = [1,2,3,4]
> > > > ... pool = multiprocessing.Pool(processes = 4)
> > > > ... pool.map(f, a)
> > > > ... pool.close()
> > > > ... pool.join()
> > > > ... channel.send('done')
> > > > ... """)
> > > > >>> Exception in thread Thread-2:
> > >
> > > Does this work if you use it in an interactive PYthon prompt
> > > without any execnet?
> > >
> > > holger
> > >
> > >
> > > > Traceback (most recent call last):
> > > >   File
> > > >
> > >
> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> > > > line 530, in __bootstrap_inner
> > > >     self.run()
> > > >   File
> > > >
> > >
> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> > > > line 483, in run
> > > >     self.__target(*self.__args, **self.__kwargs)
> > > >   File
> > > >
> > >
> "/home1/student/svilar/Development/machines/flemming/python/Lib/multiprocessing/pool.py",
> > > > line 285, in _handle_tasks
> > > >     put(task)
> > > > PicklingError: Can't pickle <type 'function'>: attribute lookup
> > > > __builtin__.function failed
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Mar 5, 2011 at 9:11 AM, holger krekel <holger at merlinux.eu>
> > > wrote:
> > > >
> > > > > On Sat, Mar 05, 2011 at 02:16 -0500, Samy Vilar wrote:
> > > > > > Is it possible to open a multiprocessing pool remotely though
> > > execnet? I
> > > > > > keep getting "PicklingError: Can't pickle <type 'function'>:
> > > attribute
> > > > > > lookup __builtin__.function failed" running on python 2.7 with
> > > execnet
> > > > > 1.0.9
> > > > >
> > > > > Can you send the code fragment you try to execute and that doesn't
> > > > > work?
> > > > >
> > > > > FWIW you can not pass code or class instances across an execnet
> > > channel.
> > > > >
> > > > > best,
> > > > > holger
> > > > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20110305/ce1a332a/attachment.html>

From holger at merlinux.eu  Mon Mar  7 10:00:26 2011
From: holger at merlinux.eu (holger krekel)
Date: Mon, 7 Mar 2011 09:00:26 +0000
Subject: [execnet-dev] help
In-Reply-To: <AANLkTik+5DjgPgopLWKKyKVxJnQvs9hL=T5hiVxs5rRK@mail.gmail.com>
References: <AANLkTi=AY_1SM_tZ0XZ1TyWdJZK0Db1Jk3NQkrLLufhD@mail.gmail.com>
	<20110305141102.GO16231@merlinux.eu>
	<AANLkTinf05A7AwkD4Ax4wU0YDAb9RmjpsNwGxKH-HtvH@mail.gmail.com>
	<20110305204851.GQ16231@merlinux.eu>
	<AANLkTiki-acLw0VnyMp2gvfNw6jgP5iY_x2uSNgS3pp4@mail.gmail.com>
	<20110305213427.GR16231@merlinux.eu>
	<AANLkTinYhA8FvtZA_f_rkzjtkXVR=OG4MVbdqVuNDeXi@mail.gmail.com>
	<AANLkTik+5DjgPgopLWKKyKVxJnQvs9hL=T5hiVxs5rRK@mail.gmail.com>
Message-ID: <20110307090026.GV16231@merlinux.eu>

On Sun, Mar 06, 2011 at 19:09 -0500, Samy Vilar wrote:
> I keep getting "can't serialize <type 'numpy.ndarray'>" numpy arrays can be
> pickled but for something reason execnet can't seem to serialize it ... and
> converting to a list kills performance both clock cycles and memory

to be clear: execnet does not use pickle for sending data through channels.
It only supports serialization of basic types like lists, dicts, ints,
strings, etc.  You can try, however, to use pickle.dumps/pickle.loads
and send the resulting string over a channel.

holger

> On Sat, Mar 5, 2011 at 9:07 PM, Samy Vilar <samy.vilar at gmail.com> wrote:
> 
> > I'll try, but do you know of a way to use execnet like pool =
> > multiprocessing.Pool(processes = 4); results =  pool.mapt(func, values)
> >
> >
> > On Sat, Mar 5, 2011 at 4:34 PM, holger krekel <holger at merlinux.eu> wrote:
> >
> >> On Sat, Mar 05, 2011 at 15:55 -0500, Samy Vilar wrote:
> >> > yep ...
> >>
> >> Without checking deeper I don't know what is going on exactly. I guess
> >> multiprocessing uses pickling and that doesn't seem to work with the way
> >> the execnet subprocess is setup.  Have you tried just using execnet to
> >> distribute your computation tasks?
> >>
> >> some examples from this page might help you if you try that:
> >>
> >>    http://codespeak.net/execnet/examples.html
> >>
> >> best,
> >> holger
> >>
> >> P.S.: please keep execnet-dev in CC because otherwise the
> >> other subscribers cannot chime in.
> >>
> >> > On Sat, Mar 5, 2011 at 3:48 PM, holger krekel <holger at merlinux.eu>
> >> wrote:
> >> >
> >> > >
> >> > >
> >> > > On Sat, Mar 05, 2011 at 14:32 -0500, Samy Vilar wrote:
> >> > > > >>> gw =
> >> > > >
> >> > >
> >> execnet.makegateway('ssh=flemming//python=/home1/student/svilar/Development/machines/flemming/python/python')
> >> > > > >>> ch = gw.remote_exec("""
> >> > > > ... import multiprocessing
> >> > > > ... def f(h):
> >> > > > ...     print h
> >> > > > ... a = [1,2,3,4]
> >> > > > ... pool = multiprocessing.Pool(processes = 4)
> >> > > > ... pool.map(f, a)
> >> > > > ... pool.close()
> >> > > > ... pool.join()
> >> > > > ... channel.send('done')
> >> > > > ... """)
> >> > > > >>> Exception in thread Thread-2:
> >> > >
> >> > > Does this work if you use it in an interactive PYthon prompt
> >> > > without any execnet?
> >> > >
> >> > > holger
> >> > >
> >> > >
> >> > > > Traceback (most recent call last):
> >> > > >   File
> >> > > >
> >> > >
> >> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> >> > > > line 530, in __bootstrap_inner
> >> > > >     self.run()
> >> > > >   File
> >> > > >
> >> > >
> >> "/home1/student/svilar/Development/machines/flemming/python/Lib/threading.py",
> >> > > > line 483, in run
> >> > > >     self.__target(*self.__args, **self.__kwargs)
> >> > > >   File
> >> > > >
> >> > >
> >> "/home1/student/svilar/Development/machines/flemming/python/Lib/multiprocessing/pool.py",
> >> > > > line 285, in _handle_tasks
> >> > > >     put(task)
> >> > > > PicklingError: Can't pickle <type 'function'>: attribute lookup
> >> > > > __builtin__.function failed
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sat, Mar 5, 2011 at 9:11 AM, holger krekel <holger at merlinux.eu>
> >> > > wrote:
> >> > > >
> >> > > > > On Sat, Mar 05, 2011 at 02:16 -0500, Samy Vilar wrote:
> >> > > > > > Is it possible to open a multiprocessing pool remotely though
> >> > > execnet? I
> >> > > > > > keep getting "PicklingError: Can't pickle <type 'function'>:
> >> > > attribute
> >> > > > > > lookup __builtin__.function failed" running on python 2.7 with
> >> > > execnet
> >> > > > > 1.0.9
> >> > > > >
> >> > > > > Can you send the code fragment you try to execute and that doesn't
> >> > > > > work?
> >> > > > >
> >> > > > > FWIW you can not pass code or class instances across an execnet
> >> > > channel.
> >> > > > >
> >> > > > > best,
> >> > > > > holger
> >> > > > >
> >> > >
> >>
> >
> >