From holger at merlinux.eu  Tue Nov  3 21:52:41 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 3 Nov 2009 21:52:41 +0100
Subject: [execnet-dev] execnet-1.0.0b1 released: now with Python3/Jython
	support
Message-ID: <20091103205241.GO29141@trillke.net>

Hi everybody!  

i just uploaded a first public release of execnet, 
namely 1.0.0b1 to PyPI.   Grab it with easy_install/pip
and your favourite Python environment (no setuptools or 
distribute required).

execnet allows to ad-hoc instantiate local and remote 
Python interpreters, currently Python2.4, 2.5, 2.6 and
(new) also Python 3.1, Jython and PyPy-c.  It works 
on Windows, Linux and OSX and is licensed under
the GPL V2 or later. 

With execnet you can instantiate "gateways" between Python processes,
use "remote_exec" code execution and "channels" to perform 
structured data communication.  See here for more info:

    http://codespeak.net/execnet

Here is a list of changes from the rather internal 
1.0.0alpha2 release:

* added new examples for NumPy, Jython, IronPython
* improved documentation
* include apipkg.py for lazy-importing
* integrated new serializer code from Benjamin Peterson
* improved support for Jython-2.5.1

have fun and let me know if you have issues,

holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From holger at merlinux.eu  Wed Nov  4 22:49:07 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 4 Nov 2009 22:49:07 +0100
Subject: [execnet-dev] 1.0.0b2 with some bug fixes
Message-ID: <20091104214907.GX29141@trillke.net>

Hi, 

I just released an execnet-1.0.0b2 to PyPI with some fixes, 
here is the changelog: 

1.0.0b2 to 1.0.0b1
--------------------------------

* make internal protocols more robust against serialization failures

* fix a seralization bug with nested tuples containing empty tuples
  (thanks to ronny for discovering it)

* setting the environment variable EXECNET_DEBUG will generate per
  process trace-files for debugging

best,
holger

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


From japerk at gmail.com  Fri Nov  6 17:05:56 2009
From: japerk at gmail.com (Jacob Perkins)
Date: Fri, 6 Nov 2009 08:05:56 -0800
Subject: [execnet-dev] closing channels and gateways
Message-ID: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>

Do channels or gateways need to be explicitly closed before a script
finishes?

When I registered an atexit function to close them (calls close() on each
channel, then exit() on each gateway), I get the following:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 979, in <module>
  File "<string>", line 688, in serve
  File "<string>", line 717, in executetask
  File "<string>", line 384, in close
  File "<string>", line 641, in _send
  File "<string>", line 185, in writeto
  File "<string>", line 884, in save
  File "<string>", line 161, in write
ValueError: I/O operation on closed file

Thanks (and huge thanks for creating execnet, it's simpler and far more
efficient for my purposes than discoproject)

Jacob

-- 
http://streamhacker.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091106/bd37647d/attachment.html>

From holger at merlinux.eu  Fri Nov  6 18:42:38 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 6 Nov 2009 18:42:38 +0100
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
Message-ID: <20091106174238.GK29141@trillke.net>

Hi Jacob, 

On Fri, Nov 06, 2009 at 08:05 -0800, Jacob Perkins wrote:
> Do channels or gateways need to be explicitly closed before a script
> finishes?

Usually not.  But automatic teardown is a bit involved, tied to GC and __del__,
not tested yet for all situations.  

> When I registered an atexit function to close them (calls close() on each
> channel, then exit() on each gateway), I get the following:
> 
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "<string>", line 979, in <module>
>   File "<string>", line 688, in serve
>   File "<string>", line 717, in executetask
>   File "<string>", line 384, in close
>   File "<string>", line 641, in _send
>   File "<string>", line 185, in writeto
>   File "<string>", line 884, in save
>   File "<string>", line 161, in write
> ValueError: I/O operation on closed file

Is this execnet-1.0.0b2? (see execnet.__version__). 
Is it part of a "RemoteError", i.e. a traceback-representation from remote?

This traceback comes from a process that finished executing 
a "remote_exec(source)" source code.  It tries to send
a "close" message but the IO connection is already dead. 
This shouldn't happen in an automatic teardown, i'd say. 
So it's a bug - if you somehow find a script to reproduce it
that'd be very helpful.

With http://codespeak.net/execnet/trunk/basics.html#debugging-execnet
you can get some trace-files of the several processes to see
what is going on. 
 
> Thanks (and huge thanks for creating execnet, it's simpler and far more
> efficient for my purposes than discoproject)

nice to hear!  What's your use case if i may ask? 

best,
holger

--
twitter: http://twitter.com/hpk42
contracting: http://merlinux.eu 


From japerk at gmail.com  Fri Nov  6 19:41:10 2009
From: japerk at gmail.com (Jacob Perkins)
Date: Fri, 6 Nov 2009 10:41:10 -0800
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <20091106174238.GK29141@trillke.net>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
	<20091106174238.GK29141@trillke.net>
Message-ID: <bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>

Hi Holger,

On Fri, Nov 6, 2009 at 9:42 AM, holger krekel <holger at merlinux.eu> wrote:

> Hi Jacob,
>
> On Fri, Nov 06, 2009 at 08:05 -0800, Jacob Perkins wrote:
> > Do channels or gateways need to be explicitly closed before a script
> > finishes?
>
> Usually not.  But automatic teardown is a bit involved, tied to GC and
> __del__,
> not tested yet for all situations.
>
> > When I registered an atexit function to close them (calls close() on each
> > channel, then exit() on each gateway), I get the following:
> >
> > Traceback (most recent call last):
> >   File "<string>", line 1, in <module>
> >   File "<string>", line 979, in <module>
> >   File "<string>", line 688, in serve
> >   File "<string>", line 717, in executetask
> >   File "<string>", line 384, in close
> >   File "<string>", line 641, in _send
> >   File "<string>", line 185, in writeto
> >   File "<string>", line 884, in save
> >   File "<string>", line 161, in write
> > ValueError: I/O operation on closed file
>
> Is this execnet-1.0.0b2? (see execnet.__version__).
>

Yes


> Is it part of a "RemoteError", i.e. a traceback-representation from remote?
>

I believe so, I got multiple tracebacks at the end of my script, so I
believe I got one from each gateway (I had 1 channel per gateway, and 10
gateways open).


>
> This traceback comes from a process that finished executing
> a "remote_exec(source)" source code.  It tries to send
> a "close" message but the IO connection is already dead.
> This shouldn't happen in an automatic teardown, i'd say.
> So it's a bug - if you somehow find a script to reproduce it
> that'd be very helpful.
>

The basic remote_exec is the following

for fname in channel:
    # open file, do stuff
    channel.send(fname)

Then in the main script I do the following (after opening gateways,
channels, and registering atexit function)

multi = execnet.MultiChannel(channels)
queue = multi.make_receive_queue()

for i in range(total): # total is the number of files sent to channels
    channel, fname = queue.get()


>
> With http://codespeak.net/execnet/trunk/basics.html#debugging-execnet
> you can get some trace-files of the several processes to see
> what is going on.
>

Ok, I'll run my script with EXECNET_DEBUG.


>
> > Thanks (and huge thanks for creating execnet, it's simpler and far more
> > efficient for my purposes than discoproject)
>
> nice to hear!  What's your use case if i may ask?
>

I'm using NLTK to do part of speech tagging and phrase chunking, so I start
a number of execnet nodes, send each two pickled NLTK objects, and then send
the nodes files to process. Since the NLTK objects can take a couple seconds
to unpickle, this is much more efficient than discoproject because the
unpickling only happens once at initialization, instead of once for every
file.


>
> best,
> holger
>
> --
> twitter: http://twitter.com/hpk42
> contracting: http://merlinux.eu
>



-- 
http://streamhacker.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091106/9c0eb7d5/attachment.html>

From japerk at gmail.com  Fri Nov  6 21:01:30 2009
From: japerk at gmail.com (Jacob Perkins)
Date: Fri, 6 Nov 2009 12:01:30 -0800
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
	<20091106174238.GK29141@trillke.net>
	<bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
Message-ID: <bf2bf0560911061201i75fd92u127637e44681849a@mail.gmail.com>

Hi Holger,

Following up after running my script w/ EXECNET_DEBUG, the trace file
contains a bunch of "sent" lines, followed by a bunch of "received" lines,
and ends with 10 "closing IO" lines. The traceback below happens once for
each gateway (or channel), after my script prints it's final message. So I
assume this happens in the atexit function, trying to close either a gateway
or channel that had already been closed automatically. Seems like the
easiest thing for me to do is simply to not bother with the atexit cleanup,
and let execnet take care of closing everything down.

Jacob

On Fri, Nov 6, 2009 at 10:41 AM, Jacob Perkins <japerk at gmail.com> wrote:

> Hi Holger,
>
> On Fri, Nov 6, 2009 at 9:42 AM, holger krekel <holger at merlinux.eu> wrote:
>
>> Hi Jacob,
>>
>> On Fri, Nov 06, 2009 at 08:05 -0800, Jacob Perkins wrote:
>> > Do channels or gateways need to be explicitly closed before a script
>> > finishes?
>>
>> Usually not.  But automatic teardown is a bit involved, tied to GC and
>> __del__,
>> not tested yet for all situations.
>>
>> > When I registered an atexit function to close them (calls close() on
>> each
>> > channel, then exit() on each gateway), I get the following:
>> >
>> > Traceback (most recent call last):
>> >   File "<string>", line 1, in <module>
>> >   File "<string>", line 979, in <module>
>> >   File "<string>", line 688, in serve
>> >   File "<string>", line 717, in executetask
>> >   File "<string>", line 384, in close
>> >   File "<string>", line 641, in _send
>> >   File "<string>", line 185, in writeto
>> >   File "<string>", line 884, in save
>> >   File "<string>", line 161, in write
>> > ValueError: I/O operation on closed file
>>
>> Is this execnet-1.0.0b2? (see execnet.__version__).
>>
>
> Yes
>
>
>> Is it part of a "RemoteError", i.e. a traceback-representation from
>> remote?
>>
>
> I believe so, I got multiple tracebacks at the end of my script, so I
> believe I got one from each gateway (I had 1 channel per gateway, and 10
> gateways open).
>
>
>>
>> This traceback comes from a process that finished executing
>> a "remote_exec(source)" source code.  It tries to send
>> a "close" message but the IO connection is already dead.
>> This shouldn't happen in an automatic teardown, i'd say.
>> So it's a bug - if you somehow find a script to reproduce it
>> that'd be very helpful.
>>
>
> The basic remote_exec is the following
>
> for fname in channel:
>     # open file, do stuff
>     channel.send(fname)
>
> Then in the main script I do the following (after opening gateways,
> channels, and registering atexit function)
>
> multi = execnet.MultiChannel(channels)
> queue = multi.make_receive_queue()
>
> for i in range(total): # total is the number of files sent to channels
>     channel, fname = queue.get()
>
>
>>
>> With http://codespeak.net/execnet/trunk/basics.html#debugging-execnet
>> you can get some trace-files of the several processes to see
>> what is going on.
>>
>
> Ok, I'll run my script with EXECNET_DEBUG.
>
>
>>
>> > Thanks (and huge thanks for creating execnet, it's simpler and far more
>> > efficient for my purposes than discoproject)
>>
>> nice to hear!  What's your use case if i may ask?
>>
>
> I'm using NLTK to do part of speech tagging and phrase chunking, so I start
> a number of execnet nodes, send each two pickled NLTK objects, and then send
> the nodes files to process. Since the NLTK objects can take a couple seconds
> to unpickle, this is much more efficient than discoproject because the
> unpickling only happens once at initialization, instead of once for every
> file.
>
>
>>
>> best,
>> holger
>>
>> --
>> twitter: http://twitter.com/hpk42
>> contracting: http://merlinux.eu
>>
>
>
>
> --
> http://streamhacker.com/
>



-- 
http://streamhacker.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091106/ab3b646c/attachment.html>

From holger at merlinux.eu  Sun Nov  8 21:01:47 2009
From: holger at merlinux.eu (holger krekel)
Date: Sun, 8 Nov 2009 21:01:47 +0100
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <bf2bf0560911061201i75fd92u127637e44681849a@mail.gmail.com>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
	<20091106174238.GK29141@trillke.net>
	<bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
	<bf2bf0560911061201i75fd92u127637e44681849a@mail.gmail.com>
Message-ID: <20091108200147.GY29141@trillke.net>

Hi Jacob,

On Fri, Nov 06, 2009 at 12:01 -0800, Jacob Perkins wrote:
> Hi Holger,
> 
> Following up after running my script w/ EXECNET_DEBUG, the trace file
> contains a bunch of "sent" lines, followed by a bunch of "received" lines,
> and ends with 10 "closing IO" lines. The traceback below happens once for
> each gateway (or channel), after my script prints it's final message. So I
> assume this happens in the atexit function, trying to close either a gateway
> or channel that had already been closed automatically. Seems like the
> easiest thing for me to do is simply to not bother with the atexit cleanup,
> and let execnet take care of closing everything down.

Makes sense but the issue probably persists, i am afraid. This example here:

    #./x.py 
    import execnet
    execnet.PopenGateway("python").remote_exec("")

if run with "python x.py" still often causes the traceback you showed.  
This is related to improper teardown. Needs further investigation ...

best,
holger

> Jacob
> 
> On Fri, Nov 6, 2009 at 10:41 AM, Jacob Perkins <japerk at gmail.com> wrote:
> 
> > Hi Holger,
> >
> > On Fri, Nov 6, 2009 at 9:42 AM, holger krekel <holger at merlinux.eu> wrote:
> >
> >> Hi Jacob,
> >>
> >> On Fri, Nov 06, 2009 at 08:05 -0800, Jacob Perkins wrote:
> >> > Do channels or gateways need to be explicitly closed before a script
> >> > finishes?
> >>
> >> Usually not.  But automatic teardown is a bit involved, tied to GC and
> >> __del__,
> >> not tested yet for all situations.
> >>
> >> > When I registered an atexit function to close them (calls close() on
> >> each
> >> > channel, then exit() on each gateway), I get the following:
> >> >
> >> > Traceback (most recent call last):
> >> >   File "<string>", line 1, in <module>
> >> >   File "<string>", line 979, in <module>
> >> >   File "<string>", line 688, in serve
> >> >   File "<string>", line 717, in executetask
> >> >   File "<string>", line 384, in close
> >> >   File "<string>", line 641, in _send
> >> >   File "<string>", line 185, in writeto
> >> >   File "<string>", line 884, in save
> >> >   File "<string>", line 161, in write
> >> > ValueError: I/O operation on closed file
> >>
> >> Is this execnet-1.0.0b2? (see execnet.__version__).
> >>
> >
> > Yes
> >
> >
> >> Is it part of a "RemoteError", i.e. a traceback-representation from
> >> remote?
> >>
> >
> > I believe so, I got multiple tracebacks at the end of my script, so I
> > believe I got one from each gateway (I had 1 channel per gateway, and 10
> > gateways open).
> >
> >
> >>
> >> This traceback comes from a process that finished executing
> >> a "remote_exec(source)" source code.  It tries to send
> >> a "close" message but the IO connection is already dead.
> >> This shouldn't happen in an automatic teardown, i'd say.
> >> So it's a bug - if you somehow find a script to reproduce it
> >> that'd be very helpful.
> >>
> >
> > The basic remote_exec is the following
> >
> > for fname in channel:
> >     # open file, do stuff
> >     channel.send(fname)
> >
> > Then in the main script I do the following (after opening gateways,
> > channels, and registering atexit function)
> >
> > multi = execnet.MultiChannel(channels)
> > queue = multi.make_receive_queue()
> >
> > for i in range(total): # total is the number of files sent to channels
> >     channel, fname = queue.get()
> >
> >
> >>
> >> With http://codespeak.net/execnet/trunk/basics.html#debugging-execnet
> >> you can get some trace-files of the several processes to see
> >> what is going on.
> >>
> >
> > Ok, I'll run my script with EXECNET_DEBUG.
> >
> >
> >>
> >> > Thanks (and huge thanks for creating execnet, it's simpler and far more
> >> > efficient for my purposes than discoproject)
> >>
> >> nice to hear!  What's your use case if i may ask?
> >>
> >
> > I'm using NLTK to do part of speech tagging and phrase chunking, so I start
> > a number of execnet nodes, send each two pickled NLTK objects, and then send
> > the nodes files to process. Since the NLTK objects can take a couple seconds
> > to unpickle, this is much more efficient than discoproject because the
> > unpickling only happens once at initialization, instead of once for every
> > file.
> >
> >
> >>
> >> best,
> >> holger
> >>
> >> --
> >> twitter: http://twitter.com/hpk42
> >> contracting: http://merlinux.eu
> >>
> >
> >
> >
> > --
> > http://streamhacker.com/
> >
> 
> 
> 
> -- 
> http://streamhacker.com/

-- 


From holger at merlinux.eu  Sun Nov  8 21:04:30 2009
From: holger at merlinux.eu (holger krekel)
Date: Sun, 8 Nov 2009 21:04:30 +0100
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
	<20091106174238.GK29141@trillke.net>
	<bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
Message-ID: <20091108200430.GZ29141@trillke.net>

Hi Jacob,

On Fri, Nov 06, 2009 at 10:41 -0800, Jacob Perkins wrote:
> > Is it part of a "RemoteError", i.e. a traceback-representation from remote?
> >
> 
> I believe so, I got multiple tracebacks at the end of my script, so I
> believe I got one from each gateway (I had 1 channel per gateway, and 10
> gateways open).

ok, still wondering a bit how this traceback printing makes it
to the main process - seems like subprocess routes it through
when a "poll()" operation is done on an Popen object? 

[...]
> > nice to hear!  What's your use case if i may ask?
> >
> 
> I'm using NLTK to do part of speech tagging and phrase chunking, so I start
> a number of execnet nodes, send each two pickled NLTK objects, and then send
> the nodes files to process. Since the NLTK objects can take a couple seconds
> to unpickle, this is much more efficient than discoproject because the
> unpickling only happens once at initialization, instead of once for every
> file.

I see.  Please don't hesitate to feedback on your experiences and 
making suggestions - i am very interested in the upcoming weeks to
improve execnet and also build a higher level lib on top of it.

cheers,
holger


From japerk at gmail.com  Sun Nov  8 21:49:17 2009
From: japerk at gmail.com (Jacob Perkins)
Date: Sun, 8 Nov 2009 12:49:17 -0800
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <20091108200430.GZ29141@trillke.net>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
	<20091106174238.GK29141@trillke.net>
	<bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
	<20091108200430.GZ29141@trillke.net>
Message-ID: <bf2bf0560911081249n6fc6b5ebncb08b04603ea97fc@mail.gmail.com>

Hi Holger,

Just in case it matters, the nodes are SSHGateways started with
execnet.makegateway('ssh=%s//chdir=%s//nice=20' % (host, cwd)).

On Sun, Nov 8, 2009 at 12:04 PM, holger krekel <holger at merlinux.eu> wrote:

> Hi Jacob,
>
> On Fri, Nov 06, 2009 at 10:41 -0800, Jacob Perkins wrote:
> > > Is it part of a "RemoteError", i.e. a traceback-representation from
> remote?
> > >
> >
> > I believe so, I got multiple tracebacks at the end of my script, so I
> > believe I got one from each gateway (I had 1 channel per gateway, and 10
> > gateways open).
>
> ok, still wondering a bit how this traceback printing makes it
> to the main process - seems like subprocess routes it through
> when a "poll()" operation is done on an Popen object?
>
> [...]
> > > nice to hear!  What's your use case if i may ask?
> > >
> >
> > I'm using NLTK to do part of speech tagging and phrase chunking, so I
> start
> > a number of execnet nodes, send each two pickled NLTK objects, and then
> send
> > the nodes files to process. Since the NLTK objects can take a couple
> seconds
> > to unpickle, this is much more efficient than discoproject because the
> > unpickling only happens once at initialization, instead of once for every
> > file.
>
> I see.  Please don't hesitate to feedback on your experiences and
> making suggestions - i am very interested in the upcoming weeks to
> improve execnet and also build a higher level lib on top of it.


A map-reduce library on top of execnet could be a good idea. However,
there's the issue whether the functions are "pure" (no external
dependencies). In my case, I depend on a lot of external libraries and
custom modules, as well as pickled objects and an external datastore
(redis), so I just make sure every node has all the code installed in a
common directory path. And all the files I'm processing are stored on a
shared GlusterFS mount point. So at this point, I'm not sure a map-reduce
abstraction layer could save me more than 10-20 loc.

Here's a some feedback based on my experience w/ execnet so far:

* The "impure" nature of execnet makes it much faster & more efficient than
discoproject given all my external dependencies. I only need to initialize &
unpickle once per gateway/channel, instead of on every map operation.

* Please be more explicit in the docs that "channel" is a global object in
the remote_exec code, but it doesn't matter what the channel is called in
the local code.

* The remote_exec channel will close if not in use. The best way I found to
keep it open is "for msg in channel:"

* I have not figured out how to shut everything down with Ctrl-C. I tried
closing everything down on a KeyboardInterrupt while waiting on a
MultiChannel receive queue, but that didn't seem to work.

* I can only open 1 channel per gateway. Not sure if that's by design, but
if so should be more explicit in the docs. I initially assumed that I would
have 1 gateway per host, then multiple channels on each gateway (1 per cpu
core). Instead, I now have 1 gateway & channel for every cpu core on each
host.

Hope that helps,
Jacob

-- 
http://streamhacker.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091108/8c60d6d1/attachment.html>

From py-dev at tolomea.com  Sun Nov  8 22:27:14 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Mon, 9 Nov 2009 08:27:14 +1100
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <bf2bf0560911081249n6fc6b5ebncb08b04603ea97fc@mail.gmail.com>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
	<20091106174238.GK29141@trillke.net>
	<bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
	<20091108200430.GZ29141@trillke.net>
	<bf2bf0560911081249n6fc6b5ebncb08b04603ea97fc@mail.gmail.com>
Message-ID: <f2d961430911081327p17258168n8b8531bf53061956@mail.gmail.com>

> * I can only open 1 channel per gateway. Not sure if that's by design, but
> if so should be more explicit in the docs. I initially assumed that I would
> have 1 gateway per host, then multiple channels on each gateway (1 per cpu
> core). Instead, I now have 1 gateway & channel for every cpu core on each
> host.

You can create a new channel by calling the newchannel function on a
gateway instance. This will return the local channel handle, to get
the remote channel handle just send the local one down an existing
channel.
I do this frequently, my system is multithreaded and for every local
thread I keep a matching remote one, each such pair of threads is
connected by their own channel. I use the original channel (the one I
got when creating the gateway) to establish the per thread channels.

G


From py-dev at tolomea.com  Mon Nov  9 03:55:21 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Mon, 9 Nov 2009 13:55:21 +1100
Subject: [execnet-dev] execnet update problems
Message-ID: <f2d961430911081855r619c445dhd04a9e1f9daf6d3f@mail.gmail.com>

I haven't been paying much attention to my execnet based project for
the last while and I am just now updating.

execnet 1.0.0b1 and 1.0.0b2 can't transport long ints


reproduction code:

import execnet
gw = execnet.SshGateway("localhost")
channel = gw.remote_exec("while True: channel.send(channel.receive())")
channel.send(1)
print channel.receive()
channel.send(3083604692L)
print channel.receive()


1.0.0alpha2:

gordonw at gohma:~$ python test.py
1
3083604692


and with 1.0.0b2:

gordonw at gohma:~$ python test.py
1
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    channel.send(3083604692L)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 382, in send
    self.gateway._send(data)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 604, in _send
    msg.writeto(self._serializer)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 148, in writeto
    serializer.save((self.msgtype, self.channelid, self.data))
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 843, in save
    self._save(obj)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 859, in _save
    dispatch(obj)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 931, in save_tuple
    self._save(item)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 859, in _save
    dispatch(obj)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 900, in save_int
    self._write_int4(i)
  File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
line 910, in _write_int4
    raise SerializationError(error)
execnet.gateway_base.SerializationError: int must be less than 2147483647
gordonw at gohma:~$ Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 979, in <module>
  File "<string>", line 688, in serve
  File "<string>", line 715, in executetask
  File "<string>", line 382, in close
  File "<string>", line 641, in _send
  File "<string>", line 185, in writeto
  File "<string>", line 884, in save
  File "<string>", line 161, in write
ValueError: I/O operation on closed file



Gordon


From py-dev at tolomea.com  Mon Nov  9 04:20:54 2009
From: py-dev at tolomea.com (Gordon Wrigley)
Date: Mon, 9 Nov 2009 14:20:54 +1100
Subject: [execnet-dev] execnet update problems
In-Reply-To: <f2d961430911081855r619c445dhd04a9e1f9daf6d3f@mail.gmail.com>
References: <f2d961430911081855r619c445dhd04a9e1f9daf6d3f@mail.gmail.com>
Message-ID: <f2d961430911081920q3b3b1d2fxfc3aa01ff45891da@mail.gmail.com>

It also can't move sets and frozensets.

On Mon, Nov 9, 2009 at 1:55 PM, Gordon Wrigley <py-dev at tolomea.com> wrote:
> I haven't been paying much attention to my execnet based project for
> the last while and I am just now updating.
>
> execnet 1.0.0b1 and 1.0.0b2 can't transport long ints
>
>
> reproduction code:
>
> import execnet
> gw = execnet.SshGateway("localhost")
> channel = gw.remote_exec("while True: channel.send(channel.receive())")
> channel.send(1)
> print channel.receive()
> channel.send(3083604692L)
> print channel.receive()
>
>
> 1.0.0alpha2:
>
> gordonw at gohma:~$ python test.py
> 1
> 3083604692
>
>
> and with 1.0.0b2:
>
> gordonw at gohma:~$ python test.py
> 1
> Traceback (most recent call last):
> ?File "test.py", line 6, in <module>
> ? ?channel.send(3083604692L)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 382, in send
> ? ?self.gateway._send(data)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 604, in _send
> ? ?msg.writeto(self._serializer)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 148, in writeto
> ? ?serializer.save((self.msgtype, self.channelid, self.data))
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 843, in save
> ? ?self._save(obj)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 859, in _save
> ? ?dispatch(obj)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 931, in save_tuple
> ? ?self._save(item)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 859, in _save
> ? ?dispatch(obj)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 900, in save_int
> ? ?self._write_int4(i)
> ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> line 910, in _write_int4
> ? ?raise SerializationError(error)
> execnet.gateway_base.SerializationError: int must be less than 2147483647
> gordonw at gohma:~$ Traceback (most recent call last):
> ?File "<string>", line 1, in <module>
> ?File "<string>", line 979, in <module>
> ?File "<string>", line 688, in serve
> ?File "<string>", line 715, in executetask
> ?File "<string>", line 382, in close
> ?File "<string>", line 641, in _send
> ?File "<string>", line 185, in writeto
> ?File "<string>", line 884, in save
> ?File "<string>", line 161, in write
> ValueError: I/O operation on closed file
>
>
>
> Gordon
>


From holger at merlinux.eu  Mon Nov  9 09:45:33 2009
From: holger at merlinux.eu (holger krekel)
Date: Mon, 9 Nov 2009 09:45:33 +0100
Subject: [execnet-dev] closing channels and gateways
In-Reply-To: <bf2bf0560911081249n6fc6b5ebncb08b04603ea97fc@mail.gmail.com>
References: <bf2bf0560911060805o1f7ad9c8nff5b7c05932cca0e@mail.gmail.com>
	<20091106174238.GK29141@trillke.net>
	<bf2bf0560911061041h3b16991du54028d1272d065ba@mail.gmail.com>
	<20091108200430.GZ29141@trillke.net>
	<bf2bf0560911081249n6fc6b5ebncb08b04603ea97fc@mail.gmail.com>
Message-ID: <20091109084533.GB29141@trillke.net>

Hi Jacob,

On Sun, Nov 08, 2009 at 12:49 -0800, Jacob Perkins wrote:
> Hi Holger,
> 
> Just in case it matters, the nodes are SSHGateways started with
> execnet.makegateway('ssh=%s//chdir=%s//nice=20' % (host, cwd)).

good to know. 

> On Sun, Nov 8, 2009 at 12:04 PM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hi Jacob,
> >
> > On Fri, Nov 06, 2009 at 10:41 -0800, Jacob Perkins wrote:
> > > > Is it part of a "RemoteError", i.e. a traceback-representation from
> > remote?
> > > >
> > >
> > > I believe so, I got multiple tracebacks at the end of my script, so I
> > > believe I got one from each gateway (I had 1 channel per gateway, and 10
> > > gateways open).
> >
> > ok, still wondering a bit how this traceback printing makes it
> > to the main process - seems like subprocess routes it through
> > when a "poll()" operation is done on an Popen object?
> >
> > [...]
> > > > nice to hear!  What's your use case if i may ask?
> > > >
> > >
> > > I'm using NLTK to do part of speech tagging and phrase chunking, so I
> > start
> > > a number of execnet nodes, send each two pickled NLTK objects, and then
> > send
> > > the nodes files to process. Since the NLTK objects can take a couple
> > seconds
> > > to unpickle, this is much more efficient than discoproject because the
> > > unpickling only happens once at initialization, instead of once for every
> > > file.
> >
> > I see.  Please don't hesitate to feedback on your experiences and
> > making suggestions - i am very interested in the upcoming weeks to
> > improve execnet and also build a higher level lib on top of it.
> 
> 
> A map-reduce library on top of execnet could be a good idea. However,
> there's the issue whether the functions are "pure" (no external
> dependencies). In my case, I depend on a lot of external libraries and
> custom modules, as well as pickled objects and an external datastore
> (redis), so I just make sure every node has all the code installed in a
> common directory path. And all the files I'm processing are stored on a
> shared GlusterFS mount point. So at this point, I'm not sure a map-reduce
> abstraction layer could save me more than 10-20 loc.

indeed the setup of the nodes is the actual work.  Many projects and people 
just use pickling which has all sorts of issues and presumes prior installation. 
In any case, for distributed testing/py.test i plan to write code on top of execnet 
that cares for setting up nodes, i.e. prepares a virtualenv or buildout environment 
and install dependencies before sending tests.  Would this make sense for you to 
have outside py.test, i.e. on some top-of-execnet lib? 

> Here's a some feedback based on my experience w/ execnet so far:
> 
> * The "impure" nature of execnet makes it much faster & more efficient than
> discoproject given all my external dependencies. I only need to initialize &
> unpickle once per gateway/channel, instead of on every map operation.

so you utilize the statefulness of gateways - remote_exec()'s do execute 
in isolated namespaces but there is the the process global "sys.modules" - 
and it's good to hear that you consider this a feature.  

> * Please be more explicit in the docs that "channel" is a global object in
> the remote_exec code, but it doesn't matter what the channel is called in
> the local code.

Ok. 

> * The remote_exec channel will close if not in use. 

what do you mean with "not in use"? 

> The best way I found to
> keep it open is "for msg in channel:"
> 
> * I have not figured out how to shut everything down with Ctrl-C. I tried
> closing everything down on a KeyboardInterrupt while waiting on a
> MultiChannel receive queue, but that didn't seem to work.

I must admit i also don't have a verified conceptual model for
making control-c work reliably.  Or to document when it can't work.

> * I can only open 1 channel per gateway. Not sure if that's by design, but
> if so should be more explicit in the docs. I initially assumed that I would
> have 1 gateway per host, then multiple channels on each gateway (1 per cpu
> core). Instead, I now have 1 gateway & channel for every cpu core on each
> host.

ups, i realize i haven't yet documented this on the web page: 

    gateway.remote_init_threads(num):
        start up to 'num' threads for subsequent
        remote_exec() invocations to allow concurrent
        execution.

On CPython you cannot efficiently have concurrent CPU bound threads, of course.
But on Jython this should be fine. 

> Hope that helps,

am in a hurry to catch a train - hope i didn't miss on
something in the mail.

cheers,
holger


> Jacob
> 
> -- 
> http://streamhacker.com/

-- 


From holger at merlinux.eu  Mon Nov  9 16:10:55 2009
From: holger at merlinux.eu (holger krekel)
Date: Mon, 9 Nov 2009 16:10:55 +0100
Subject: [execnet-dev] execnet update problems
In-Reply-To: <f2d961430911081920q3b3b1d2fxfc3aa01ff45891da@mail.gmail.com>
References: <f2d961430911081855r619c445dhd04a9e1f9daf6d3f@mail.gmail.com>
	<f2d961430911081920q3b3b1d2fxfc3aa01ff45891da@mail.gmail.com>
Message-ID: <20091109151055.GE29141@trillke.net>

Hi Gordon,

On Mon, Nov 09, 2009 at 14:20 +1100, Gordon Wrigley wrote:
> It also can't move sets and frozensets.

yes, the new serializer is missing proper support for sets 
and longs.  I asked Benjamin for help and otherwise will 
fix it myself. 

cheers,
holger

> On Mon, Nov 9, 2009 at 1:55 PM, Gordon Wrigley <py-dev at tolomea.com> wrote:
> > I haven't been paying much attention to my execnet based project for
> > the last while and I am just now updating.
> >
> > execnet 1.0.0b1 and 1.0.0b2 can't transport long ints
> >
> >
> > reproduction code:
> >
> > import execnet
> > gw = execnet.SshGateway("localhost")
> > channel = gw.remote_exec("while True: channel.send(channel.receive())")
> > channel.send(1)
> > print channel.receive()
> > channel.send(3083604692L)
> > print channel.receive()
> >
> >
> > 1.0.0alpha2:
> >
> > gordonw at gohma:~$ python test.py
> > 1
> > 3083604692
> >
> >
> > and with 1.0.0b2:
> >
> > gordonw at gohma:~$ python test.py
> > 1
> > Traceback (most recent call last):
> > ?File "test.py", line 6, in <module>
> > ? ?channel.send(3083604692L)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 382, in send
> > ? ?self.gateway._send(data)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 604, in _send
> > ? ?msg.writeto(self._serializer)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 148, in writeto
> > ? ?serializer.save((self.msgtype, self.channelid, self.data))
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 843, in save
> > ? ?self._save(obj)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 859, in _save
> > ? ?dispatch(obj)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 931, in save_tuple
> > ? ?self._save(item)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 859, in _save
> > ? ?dispatch(obj)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 900, in save_int
> > ? ?self._write_int4(i)
> > ?File "/usr/local/lib/python2.6/dist-packages/execnet-1.0.0b2-py2.6.egg/execnet/gateway_base.py",
> > line 910, in _write_int4
> > ? ?raise SerializationError(error)
> > execnet.gateway_base.SerializationError: int must be less than 2147483647
> > gordonw at gohma:~$ Traceback (most recent call last):
> > ?File "<string>", line 1, in <module>
> > ?File "<string>", line 979, in <module>
> > ?File "<string>", line 688, in serve
> > ?File "<string>", line 715, in executetask
> > ?File "<string>", line 382, in close
> > ?File "<string>", line 641, in _send
> > ?File "<string>", line 185, in writeto
> > ?File "<string>", line 884, in save
> > ?File "<string>", line 161, in write
> > ValueError: I/O operation on closed file
> >
> >
> >
> > Gordon
> >
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev
> 

-- 


From holger at merlinux.eu  Tue Nov 10 00:46:31 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 10 Nov 2009 00:46:31 +0100
Subject: [execnet-dev] uploaded a 1.0.0b3
Message-ID: <20091109234631.GI29141@trillke.net>

Hi all, 

just uploaded a new 1.0.0b3 of execnet.
see below for the changelog.  

cheers,
holger

1.0.0b3
--------------------------------

* fix EXECNET_DEBUG to work with win32
* add support for serializing longs, sets and frozensets
  (thanks Benjamin Peterson)
* introduce remote_status() method which on the low level gives
  information about the remote side of a gateway
* disallow explicit close in remote_exec situation
* perform some more detailed tracing with EXECNET_DEBUG


From Adam.Schmalhofer at gmx.de  Wed Nov 18 12:55:42 2009
From: Adam.Schmalhofer at gmx.de (Adam)
Date: Wed, 18 Nov 2009 12:55:42 +0100
Subject: [execnet-dev] testing vim with execnet
Message-ID: <20091118125542.55c5db7f@deepthought.local>

I am trying to test a vim-plugin (blogit.vim) via execnet (and py.test). I
let vim start the SocketServer (vim -c "pyfile socketserver.py"). I can
connect to it and run different vim commands (import vim;
vim.command("%s/a/e")).

However I can't access any of vim's python objects directly. I can only
let vim manipulate them (via vim.command) and exchange strings over
vim.current.buffer. e.g.:

    >>> channel = gw.remote_exec("""
    ... import vim
    ... vim.command("py b = 42")
    ... vim.command("py vim.current.buffer[0] = str(b)")
    ... channel.send(vim.current.buffer[0])    # works: '42'
    ... channel.send(b)    # fails: NameError
    ... """)

The other direction doesn't work either. The two have uncoupled
namespaces (it seems). unpickling strings exchanged over
vim.current.buffer fails, too.

Any ideas how I can make this connection would be great.
Thanks for your time.

?Adam
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 828 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091118/7834afc0/attachment.pgp>

From holger at merlinux.eu  Wed Nov 18 16:23:47 2009
From: holger at merlinux.eu (holger krekel)
Date: Wed, 18 Nov 2009 16:23:47 +0100
Subject: [execnet-dev] testing vim with execnet
In-Reply-To: <20091118125542.55c5db7f@deepthought.local>
References: <20091118125542.55c5db7f@deepthought.local>
Message-ID: <20091118152347.GP17999@trillke.net>

Hi Adam,

On Wed, Nov 18, 2009 at 12:55 +0100, Adam wrote:
> I am trying to test a vim-plugin (blogit.vim) via execnet (and py.test). I
> let vim start the SocketServer (vim -c "pyfile socketserver.py"). I can
> connect to it and run different vim commands (import vim;
> vim.command("%s/a/e")).
> 
> However I can't access any of vim's python objects directly. I can only
> let vim manipulate them (via vim.command) and exchange strings over
> vim.current.buffer. e.g.:
> 
>     >>> channel = gw.remote_exec("""
>     ... import vim
>     ... vim.command("py b = 42")
>     ... vim.command("py vim.current.buffer[0] = str(b)")
>     ... channel.send(vim.current.buffer[0])    # works: '42'
>     ... channel.send(b)    # fails: NameError
>     ... """)
> 
> The other direction doesn't work either. The two have uncoupled
> namespaces (it seems). 

yes, true and nothing that could easily change i think. 
Maybe you could try to extract values via 'vim.eval'? 
I haven't used the vim-python-plugin myself yet, btw.

> unpickling strings exchanged over
> vim.current.buffer fails, too.

not sure what you mean - Strings should be sendable/receiveable. 
what exactly fails? 

best,
holger

> Any ideas how I can make this connection would be great.
> Thanks for your time.
> 
> ???Adam



> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev


-- 


From Adam.Schmalhofer at gmx.de  Thu Nov 19 18:29:57 2009
From: Adam.Schmalhofer at gmx.de (Adam)
Date: Thu, 19 Nov 2009 18:29:57 +0100
Subject: [execnet-dev] testing vim with execnet
In-Reply-To: <20091118152347.GP17999@trillke.net>
References: <20091118125542.55c5db7f@deepthought.local>
	<20091118152347.GP17999@trillke.net>
Message-ID: <20091119182957.1f6eab21@deepthought.local>

Hi Holger,

holger krekel wrote:

> On Wed, Nov 18, 2009 at 12:55 +0100, Adam wrote:

> > However I can't access any of vim's python objects directly. I can only
> > let vim manipulate them (via vim.command) and exchange strings over
> > vim.current.buffer. e.g.:

> yes, true and nothing that could easily change i think. 

Ok. This surprised me as socketserver has access to these. 

> Maybe you could try to extract values via 'vim.eval'? 

vim.eval gives access to vim-script but not vim's python. This gave me
the right hint for the solution (see below).

> I haven't used the vim-python-plugin myself yet, btw.

I didn't assume that. I asked here because I thought it wasn't needed
(see two above).

> > unpickling strings exchanged over vim.current.buffer fails, too.
> 
> not sure what you mean - Strings should be sendable/receiveable. what
> exactly fails? 

I thought that pickle strings where too exotic for vim.current.buffer.
Now, I really have no clue, what fails.[1] Pickling over vim.eval does
works, however.[2] Thank you for the idea. Now I'll work on sugaring it.
Problem solved.

--Adam


[1] I'm either making a stupid mistake or something really funky is
    going on. If I type this into vim directly (without the vim.command
    everything works.

    Left for the interested. Irrelevant now as [2] works well:

    >>> def test_execnet_buffer_pickle(vim_gateway):
    >>>     import pickle
    >>>     channel = vim_gateway.remote_exec(r"""
    ...         import vim
    ...         vim.command('py import pickle')
    ...         vim.command('py import vim')
    ...         vim.command('py b = pickle.dumps([1, 2, 3])')
    ...         vim.command('py if b == "": b = "empty"')
    ...         vim.command('py vim.current.buffer[0] = b.split("\n")[0]')
    ...         vim.command('py if vim.current.buffer[0] == "": vim.current.buffer[0] = "broken"')
    >>>         channel.send(vim.current.buffer[0])""")
    >>>     buf = channel.receive()
    >>>     channel.close()
    >>>     assert buf != 'broken'    # fails
    >>>     #assert '\n'.join(buf) == pickle.dumps([1, 2, 3])    # '(lp0\nI1\naI2\naI3\na.'


[2] The solution:

    >>> def test_execnet_eval_pickle(vim_gateway):
    >>>     import pickle
    >>>     channel = vim_gateway.remote_exec(r"""
    ...         import vim
    ...         vim.command('py import pickle')
    ...         vim.command('py import vim')
    ...         vim.command('py b = pickle.dumps([1, 2, 3])')
    ...         vim.command('''py vim.command("let execnet='%s'" % b)''')
    ...         b = vim.eval("execnet")
    ...         channel.send(b)""")
    >>>     buf = channel.receive()
    >>>     channel.close()
    >>>     assert buf == pickle.dumps([1, 2, 3])    # passes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 828 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091119/98948241/attachment.pgp>

From Ronny.Pfannschmidt at gmx.de  Thu Nov 19 21:29:22 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Thu, 19 Nov 2009 21:29:22 +0100
Subject: [execnet-dev] testing vim with execnet
In-Reply-To: <20091118125542.55c5db7f@deepthought.local>
References: <20091118125542.55c5db7f@deepthought.local>
Message-ID: <1258662562.3461.3.camel@localhost>

On Wed, 2009-11-18 at 12:55 +0100, Adam wrote:
> I am trying to test a vim-plugin (blogit.vim) via execnet (and py.test). I
> let vim start the SocketServer (vim -c "pyfile socketserver.py"). I can
> connect to it and run different vim commands (import vim;
> vim.command("%s/a/e")).
> 
> However I can't access any of vim's python objects directly. I can only
> let vim manipulate them (via vim.command) and exchange strings over
> vim.current.buffer. e.g.:
> 
>     >>> channel = gw.remote_exec("""
>     ... import vim
>     ... vim.command("py b = 42")
>     ... vim.command("py vim.current.buffer[0] = str(b)")
>     ... channel.send(vim.current.buffer[0])    # works: '42'
>     ... channel.send(b)    # fails: NameError
>     ... """)
> 
> The other direction doesn't work either. The two have uncoupled
> namespaces (it seems). unpickling strings exchanged over
> vim.current.buffer fails, too.
> 
> Any ideas how I can make this connection would be great.
> Thanks for your time.
> 
> ?Adam

In pida we wrote a small vim addon that started a dbus-python based
server in order to expose some of vim as a (rather crude and hackish)
dbus object.

I'd love if someone collaborated on getting that to a state for doing
more serious things wrt testing vim and vim addons.

Regards Ronny



From Adam.Schmalhofer at gmx.de  Thu Nov 19 21:56:42 2009
From: Adam.Schmalhofer at gmx.de (Adam)
Date: Thu, 19 Nov 2009 21:56:42 +0100
Subject: [execnet-dev] testing vim with execnet
In-Reply-To: <1258662562.3461.3.camel@localhost>
References: <20091118125542.55c5db7f@deepthought.local>
	<1258662562.3461.3.camel@localhost>
Message-ID: <20091119215642.6010a0c9@deepthought.local>

Ronny Pfannschmidt wrote:

> In pida we wrote a small vim addon that started a dbus-python based
> server in order to expose some of vim as a (rather crude and hackish)
> dbus object.

How extensive is the access you get?

?Adam
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 828 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091119/4bd61994/attachment.pgp>

From holger at merlinux.eu  Fri Nov 20 00:24:31 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 20 Nov 2009 00:24:31 +0100
Subject: [execnet-dev] 1.0.0b4: execnet.Group and improvements
Message-ID: <20091119232431.GU17999@trillke.net>

Hi all,

today i committed a number of interesting changes that
hopefully unify and enhance the execnet experience: 

* NEW execnet.Group() for managing groups of gateways,
  gateways have a (per-group) unique id and
  termination can be triggered per-group

* makegateway() is now the primary way to create
  gateways, other uses are deprecated

* most website examples are now automatically tested

* termination enhancements, general website/doc enhancements

I'd be very grateful to get feedback on the new API
and the improved docs and examples:

    http://codespeak.net/execnet/1.0.0b4

and 

    easy_install http://merlinux.eu/~hpk/execnet-1.0.0b4.tar.gz

I am not releasing this publically right away as it would break 
py.test aka py-1.1.0 of which i am preparing a py-1.1.1 release
which will work with execnet-1.0.0b4. Also i'd love to incorporate
any refinements to the new "Group" concept that you might have.

best,
holger


From holger at merlinux.eu  Fri Nov 20 19:48:16 2009
From: holger at merlinux.eu (holger krekel)
Date: Fri, 20 Nov 2009 19:48:16 +0100
Subject: [execnet-dev] testing vim with execnet
In-Reply-To: <20091119182957.1f6eab21@deepthought.local>
References: <20091118125542.55c5db7f@deepthought.local>
	<20091118152347.GP17999@trillke.net>
	<20091119182957.1f6eab21@deepthought.local>
Message-ID: <20091120184816.GA17999@trillke.net>

Hi Adam,

i am curious: how does the setup of the vim_gateway funcarg look like? 
And how should i best install vim-bindings on an ubuntu 9.04 machine
in order to run such code as you post? 

best,
holger

On Thu, Nov 19, 2009 at 18:29 +0100, Adam wrote:
> [1] I'm either making a stupid mistake or something really funky is
>     going on. If I type this into vim directly (without the vim.command
>     everything works.
> 
>     Left for the interested. Irrelevant now as [2] works well:
> 
>     >>> def test_execnet_buffer_pickle(vim_gateway):
>     >>>     import pickle
>     >>>     channel = vim_gateway.remote_exec(r"""
>     ...         import vim
>     ...         vim.command('py import pickle')
>     ...         vim.command('py import vim')
>     ...         vim.command('py b = pickle.dumps([1, 2, 3])')
>     ...         vim.command('py if b == "": b = "empty"')
>     ...         vim.command('py vim.current.buffer[0] = b.split("\n")[0]')
>     ...         vim.command('py if vim.current.buffer[0] == "": vim.current.buffer[0] = "broken"')
>     >>>         channel.send(vim.current.buffer[0])""")
>     >>>     buf = channel.receive()
>     >>>     channel.close()
>     >>>     assert buf != 'broken'    # fails
>     >>>     #assert '\n'.join(buf) == pickle.dumps([1, 2, 3])    # '(lp0\nI1\naI2\naI3\na.'
> 
> 
> [2] The solution:
> 
>     >>> def test_execnet_eval_pickle(vim_gateway):
>     >>>     import pickle
>     >>>     channel = vim_gateway.remote_exec(r"""
>     ...         import vim
>     ...         vim.command('py import pickle')
>     ...         vim.command('py import vim')
>     ...         vim.command('py b = pickle.dumps([1, 2, 3])')
>     ...         vim.command('''py vim.command("let execnet='%s'" % b)''')
>     ...         b = vim.eval("execnet")
>     ...         channel.send(b)""")
>     >>>     buf = channel.receive()
>     >>>     channel.close()
>     >>>     assert buf == pickle.dumps([1, 2, 3])    # passes



From Adam.Schmalhofer at gmx.de  Sun Nov 22 09:54:06 2009
From: Adam.Schmalhofer at gmx.de (Adam)
Date: Sun, 22 Nov 2009 09:54:06 +0100
Subject: [execnet-dev] testing vim with execnet
In-Reply-To: <20091120184816.GA17999@trillke.net>
References: <20091118125542.55c5db7f@deepthought.local>
	<20091118152347.GP17999@trillke.net>
	<20091119182957.1f6eab21@deepthought.local>
	<20091120184816.GA17999@trillke.net>
Message-ID: <20091122095406.78516d7e@deepthought.local>

Hi Holger,

> i am curious: how does the setup of the vim_gateway funcarg look like? 

Sure[1]. It is currently very minimalistic and the timeout is totally
paranoid. Don't hesitate if you have critics or questions.

> And how should i best install vim-bindings on an ubuntu 9.04 machine
> in order to run such code as you post? 

The bindings are included in the vim packages (vim, vim-nox, vim-full,
vim-gtk, vim-gnome) with the exception of vim-tiny. So vim users usually
already have them installed (vim-tiny doesn't even support syntax
highlighting).

--Adam


[1] 

def pytest_funcarg__vim_gateway(request):
    if not request.config.option.acceptance:
        py.test.skip('specify -A to run acceptance tests')
    if execnet is None:
        py.test.skip('Install execnet to run vim acceptance tests.')
    #skip_assert_port_available('localhost', 8888)
    vim_proc = Popen(['vim', '-c', 'pyfile socketserver.py'], stdout=PIPE)
    gw = create_socket_gateway('localhost', 8888)

    def teardown():
        try:
            channel = gw.remote_exec('import vim; vim.command("q!")')
            for i in range(90):
                if not vim_proc.poll() is None:
                    break
                time.sleep(1)
            else:
                vim_proc.kill()    # only works in python2.6+
            gw.exit()
        except:
            pass
    request.addfinalizer(teardown)
    return gw


def create_socket_gateway(host, port):
    for i in range(90):
        try:
            return execnet.SocketGateway(host, port)
        except socket.error, e:
            if e.args[0] == 111:
                # 'Conection refused': Server isn't up, yet.
                time.sleep(1)
            else:
                raise
    else:
        py.test.skip('failed to connect to vim via socketserver.')



> On Thu, Nov 19, 2009 at 18:29 +0100, Adam wrote:
> > [1] I'm either making a stupid mistake or something really funky is
> >     going on. If I type this into vim directly (without the vim.command
> >     everything works.
> > 
> >     Left for the interested. Irrelevant now as [2] works well:
> > 
> >     >>> def test_execnet_buffer_pickle(vim_gateway):
> >     >>>     import pickle
> >     >>>     channel = vim_gateway.remote_exec(r"""
> >     ...         import vim
> >     ...         vim.command('py import pickle')
> >     ...         vim.command('py import vim')
> >     ...         vim.command('py b = pickle.dumps([1, 2, 3])')
> >     ...         vim.command('py if b == "": b = "empty"')
> >     ...         vim.command('py vim.current.buffer[0] = b.split("\n")[0]')
> >     ...         vim.command('py if vim.current.buffer[0] == "":
> > vim.current.buffer[0] = "broken"')
> >     >>>         channel.send(vim.current.buffer[0])""")
> >     >>>     buf = channel.receive()
> >     >>>     channel.close()
> >     >>>     assert buf != 'broken'    # fails
> >     >>>     #assert '\n'.join(buf) == pickle.dumps([1, 2, 3])    #
> >     >>> '(lp0\nI1\naI2\naI3\na.'
> > 
> > 
> > [2] The solution:
> > 
> >     >>> def test_execnet_eval_pickle(vim_gateway):
> >     >>>     import pickle
> >     >>>     channel = vim_gateway.remote_exec(r"""
> >     ...         import vim
> >     ...         vim.command('py import pickle')
> >     ...         vim.command('py import vim')
> >     ...         vim.command('py b = pickle.dumps([1, 2, 3])')
> >     ...         vim.command('''py vim.command("let execnet='%s'" % b)''')
> >     ...         b = vim.eval("execnet")
> >     ...         channel.send(b)""")
> >     >>>     buf = channel.receive()
> >     >>>     channel.close()
> >     >>>     assert buf == pickle.dumps([1, 2, 3])    # passes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 828 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091122/dc3df860/attachment.pgp>

From Ronny.Pfannschmidt at gmx.de  Sun Nov 22 14:25:17 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Sun, 22 Nov 2009 14:25:17 +0100
Subject: [execnet-dev] testing vim with execnet
In-Reply-To: <20091119215642.6010a0c9@deepthought.local>
References: <20091118125542.55c5db7f@deepthought.local>
	<1258662562.3461.3.camel@localhost>
	<20091119215642.6010a0c9@deepthought.local>
Message-ID: <1258896317.11078.24.camel@localhost>

On Thu, 2009-11-19 at 21:56 +0100, Adam wrote:
> Ronny Pfannschmidt wrote:
> 
> > In pida we wrote a small vim addon that started a dbus-python based
> > server in order to expose some of vim as a (rather crude and hackish)
> > dbus object.
> 
> How extensive is the access you get?

As for the dbus server, it can access anything within vim cause it runs
within vim.

However the api we currently expose is rather crude and limited (its the
bare minimum to get gvim doing what we need it to do for the ide).

The goal should be to have a reasonable api for Accessing a vim
instance.

Currently its only tested in gvim
experiments on the curses based vim are yet to be conducted.

-Ronny



From Ronny.Pfannschmidt at gmx.de  Mon Nov 23 01:55:42 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Mon, 23 Nov 2009 01:55:42 +0100
Subject: [execnet-dev] libcloud support?
Message-ID: <1258937742.28787.2.camel@localhost>

hi,

i just stumbled uppon http://incubator.apache.org/libcloud/

its a python lib for managing server instances on cloud providers
the api looks a bit weird, but the rest looks helpfull

Regards Ronny



From holger at merlinux.eu  Tue Nov 24 10:06:22 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 24 Nov 2009 10:06:22 +0100
Subject: [execnet-dev] libcloud support?
In-Reply-To: <1258937742.28787.2.camel@localhost>
References: <1258937742.28787.2.camel@localhost>
Message-ID: <20091124090622.GH17999@trillke.net>

Hi Ronny,

On Mon, Nov 23, 2009 at 01:55 +0100, Ronny Pfannschmidt wrote:
> hi,
> 
> i just stumbled uppon http://incubator.apache.org/libcloud/
> 
> its a python lib for managing server instances on cloud providers
> the api looks a bit weird, but the rest looks helpfull

interesting.  It's higher level than what execnet currently
does.  Might make for a nice combination, use ligcloud
to setup and manage hosts and use execnet to dynamically work 
with them. 

best,
holger


From holger at merlinux.eu  Tue Nov 24 17:49:50 2009
From: holger at merlinux.eu (holger krekel)
Date: Tue, 24 Nov 2009 17:49:50 +0100
Subject: [execnet-dev] execnet-1.0.0 released
Message-ID: <20091124164950.GI17999@trillke.net>

Hi all,

execnet-1.0.0 (compared to 1.0.0b3) has bug fixes, new tested
examples and introduces execnet.Group for managing a dynamic
bunch of hosts.  See the improved docs

    http://codespeak.net/execnet/

and below the changelog fragment.  

cheers,
holger


1.0.0 compared to 1.0.0b3
--------------------------------

* introduce execnet.Group for managing gateway creation
  and termination.  Introduce execnet.default_group through which
  all "global" calls are routed.  cleanup gateway termination.
  All Gateways get an id through which they can be
  retrieved from a group object.

* deprecate execnet.XYZGateway in favour of direct makegateway() calls.

* refine socketserver-examples, experimentally introduce a
  way to indirectly setup a socket server ("installvia")
  through a gateway url.

* refine and automatically test documentation examples


From japerk at gmail.com  Sun Nov 29 19:05:27 2009
From: japerk at gmail.com (Jacob Perkins)
Date: Sun, 29 Nov 2009 10:05:27 -0800
Subject: [execnet-dev] nltk and execnet
Message-ID: <bf2bf0560911291005x43a0d17bq2fb4aa813b3be48@mail.gmail.com>

In honor of the 1.0.0 release, I just published an article on using NLTK
with execnet.

http://streamhacker.com/2009/11/29/distributed-nltk-execnet/

Jacob
-- 
http://streamhacker.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20091129/c6e89772/attachment.html>