From rick1 at gerkin.org  Mon Oct 14 22:40:49 2013
From: rick1 at gerkin.org (Rick Gerkin)
Date: Mon, 14 Oct 2013 13:40:49 -0700
Subject: [execnet-dev] Persistent sessions
Message-ID: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>

Is there anyway to make the contents of the interpreter persist between
calls to remote_exec?  For example,

import execnet
gw = execnet.makegateway("popen//python=jython")
ch1 = gw.remote_exec('var=2; channel.send(var)')
ch1.receive() # Prints 2.
ch2 = gw.remote_exec('channel.send(var)')
ch2.receive() # NameError: name 'var' is not defined.

I would expect the gateway to correspond to one continuous python session,
but I guess it doesn't?

Rick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20131014/e7b05594/attachment.html>

From holger at merlinux.eu  Mon Oct 14 22:47:49 2013
From: holger at merlinux.eu (holger krekel)
Date: Mon, 14 Oct 2013 20:47:49 +0000
Subject: [execnet-dev] Persistent sessions
In-Reply-To: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>
References: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>
Message-ID: <20131014204749.GZ14010@merlinux.eu>

On Mon, Oct 14, 2013 at 13:40 -0700, Rick Gerkin wrote:
> Is there anyway to make the contents of the interpreter persist between
> calls to remote_exec?  For example,
> 
> import execnet
> gw = execnet.makegateway("popen//python=jython")
> ch1 = gw.remote_exec('var=2; channel.send(var)')
> ch1.receive() # Prints 2.
> ch2 = gw.remote_exec('channel.send(var)')
> ch2.receive() # NameError: name 'var' is not defined.
> 
> I would expect the gateway to correspond to one continuous python session,
> but I guess it doesn't?

Each remote_exec() has a separate namespace.  There is no execnet-offered
"sharing" space that survives calls.  We could think about something.
Meanwhile you can stick things on an imported module, for example
"sys" or "os" if you don't have own ones.  Or write a little
support code that puts an artifical module into sys.modules,
like this:

    def get_shared_data():
        name = "shared_data"
        try:
            return sys.modules[name]
        except KeyError:
            mod = sys.modules[name] = types.ModuleType(name)
            l.append(mod)
            return mod

You can then use the result of get_shared_data() to store/retrieve
attributes across sessions.

holger

From dimaqq at gmail.com  Mon Oct 14 22:45:54 2013
From: dimaqq at gmail.com (Dima Tisnek)
Date: Mon, 14 Oct 2013 13:45:54 -0700
Subject: [execnet-dev] Persistent sessions
In-Reply-To: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>
References: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>
Message-ID: <CAGGBzXKd60gcs9Ludk42GmBCahEq4N_8L_w69vTrmZksvKeK-Q@mail.gmail.com>

rick, I believe stateless server is by design.

the workaround I used was to have the server ping the client and
client send next command within the context of responding to this ping
message.

there might be a more elegant solution, perhaps involving generators.
in the end it really depends what you are trying to do.

On 14 October 2013 13:40, Rick Gerkin <rick1 at gerkin.org> wrote:
> Is there anyway to make the contents of the interpreter persist between
> calls to remote_exec?  For example,
>
> import execnet
> gw = execnet.makegateway("popen//python=jython")
> ch1 = gw.remote_exec('var=2; channel.send(var)')
> ch1.receive() # Prints 2.
> ch2 = gw.remote_exec('channel.send(var)')
> ch2.receive() # NameError: name 'var' is not defined.
>
> I would expect the gateway to correspond to one continuous python session,
> but I guess it doesn't?
>
> Rick
>
>
>
>
>
>
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at python.org
> https://mail.python.org/mailman/listinfo/execnet-dev
>

From rick1 at gerkin.org  Mon Oct 14 22:58:28 2013
From: rick1 at gerkin.org (Rick Gerkin)
Date: Mon, 14 Oct 2013 13:58:28 -0700
Subject: [execnet-dev] Persistent sessions
In-Reply-To: <20131014204749.GZ14010@merlinux.eu>
References: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>
 <20131014204749.GZ14010@merlinux.eu>
Message-ID: <CAPtMRDXnaJNrNNjOmpitzCipT3wpeSu5MqKGMoBWN8GbiZFUjg@mail.gmail.com>

Setting attributes of imported modules should work fine.  Thanks!

There is also this:
http://codespeak.net/execnet/example/test_info.html#a-simple-command-loop-pattern
which won't work in my case because the module I would need to send over is
jython only (invoking execnet from Cpython), but otherwise seems useful.


On Mon, Oct 14, 2013 at 1:47 PM, holger krekel <holger at merlinux.eu> wrote:

> On Mon, Oct 14, 2013 at 13:40 -0700, Rick Gerkin wrote:
> > Is there anyway to make the contents of the interpreter persist between
> > calls to remote_exec?  For example,
> >
> > import execnet
> > gw = execnet.makegateway("popen//python=jython")
> > ch1 = gw.remote_exec('var=2; channel.send(var)')
> > ch1.receive() # Prints 2.
> > ch2 = gw.remote_exec('channel.send(var)')
> > ch2.receive() # NameError: name 'var' is not defined.
> >
> > I would expect the gateway to correspond to one continuous python
> session,
> > but I guess it doesn't?
>
> Each remote_exec() has a separate namespace.  There is no execnet-offered
> "sharing" space that survives calls.  We could think about something.
> Meanwhile you can stick things on an imported module, for example
> "sys" or "os" if you don't have own ones.  Or write a little
> support code that puts an artifical module into sys.modules,
> like this:
>
>     def get_shared_data():
>         name = "shared_data"
>         try:
>             return sys.modules[name]
>         except KeyError:
>             mod = sys.modules[name] = types.ModuleType(name)
>             l.append(mod)
>             return mod
>
> You can then use the result of get_shared_data() to store/retrieve
> attributes across sessions.
>
> holger
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20131014/2a446c50/attachment.html>

From holger at merlinux.eu  Mon Oct 14 23:03:13 2013
From: holger at merlinux.eu (holger krekel)
Date: Mon, 14 Oct 2013 21:03:13 +0000
Subject: [execnet-dev] Persistent sessions
In-Reply-To: <CAPtMRDXnaJNrNNjOmpitzCipT3wpeSu5MqKGMoBWN8GbiZFUjg@mail.gmail.com>
References: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>
 <20131014204749.GZ14010@merlinux.eu>
 <CAPtMRDXnaJNrNNjOmpitzCipT3wpeSu5MqKGMoBWN8GbiZFUjg@mail.gmail.com>
Message-ID: <20131014210313.GB14010@merlinux.eu>

On Mon, Oct 14, 2013 at 13:58 -0700, Rick Gerkin wrote:
> Setting attributes of imported modules should work fine.  Thanks!
> 
> There is also this:
> http://codespeak.net/execnet/example/test_info.html#a-simple-command-loop-pattern
> which won't work in my case because the module I would need to send over is
> jython only (invoking execnet from Cpython), but otherwise seems useful.

I recently had Alfredo (CCed) also wondering about this snippet.  It's really
meant as a very small example, certainly needs some more flesh to be
convenient.  Not sure what kind of solution he is using for ceph-deploy now.

holger

> 
> On Mon, Oct 14, 2013 at 1:47 PM, holger krekel <holger at merlinux.eu> wrote:
> 
> > On Mon, Oct 14, 2013 at 13:40 -0700, Rick Gerkin wrote:
> > > Is there anyway to make the contents of the interpreter persist between
> > > calls to remote_exec?  For example,
> > >
> > > import execnet
> > > gw = execnet.makegateway("popen//python=jython")
> > > ch1 = gw.remote_exec('var=2; channel.send(var)')
> > > ch1.receive() # Prints 2.
> > > ch2 = gw.remote_exec('channel.send(var)')
> > > ch2.receive() # NameError: name 'var' is not defined.
> > >
> > > I would expect the gateway to correspond to one continuous python
> > session,
> > > but I guess it doesn't?
> >
> > Each remote_exec() has a separate namespace.  There is no execnet-offered
> > "sharing" space that survives calls.  We could think about something.
> > Meanwhile you can stick things on an imported module, for example
> > "sys" or "os" if you don't have own ones.  Or write a little
> > support code that puts an artifical module into sys.modules,
> > like this:
> >
> >     def get_shared_data():
> >         name = "shared_data"
> >         try:
> >             return sys.modules[name]
> >         except KeyError:
> >             mod = sys.modules[name] = types.ModuleType(name)
> >             l.append(mod)
> >             return mod
> >
> > You can then use the result of get_shared_data() to store/retrieve
> > attributes across sessions.
> >
> > holger
> >

From alfredodeza at gmail.com  Tue Oct 15 15:32:54 2013
From: alfredodeza at gmail.com (Alfredo Deza)
Date: Tue, 15 Oct 2013 09:32:54 -0400
Subject: [execnet-dev] Persistent sessions
In-Reply-To: <20131014210313.GB14010@merlinux.eu>
References: <CAPtMRDVU+UFq2pyNv6PX20uSpgziH5EjgnT+QQq662Q4Fu2Wmg@mail.gmail.com>
 <20131014204749.GZ14010@merlinux.eu>
 <CAPtMRDXnaJNrNNjOmpitzCipT3wpeSu5MqKGMoBWN8GbiZFUjg@mail.gmail.com>
 <20131014210313.GB14010@merlinux.eu>
Message-ID: <9292E9B3-39F8-4ECB-BF67-27A803EC6750@gmail.com>


On Oct 14, 2013, at 5:03 PM, holger krekel <holger at merlinux.eu> wrote:

> On Mon, Oct 14, 2013 at 13:58 -0700, Rick Gerkin wrote:
>> Setting attributes of imported modules should work fine.  Thanks!
>> 
>> There is also this:
>> http://codespeak.net/execnet/example/test_info.html#a-simple-command-loop-pattern
>> which won't work in my case because the module I would need to send over is
>> jython only (invoking execnet from Cpython), but otherwise seems useful.
> 
> I recently had Alfredo (CCed) also wondering about this snippet.  It's really
> meant as a very small example, certainly needs some more flesh to be
> convenient.  Not sure what kind of solution he is using for ceph-deploy now.
> 

The case of ceph-deploy (a tool to install/configure Ceph on remote hosts) is kind of unique. There are
like 6 or 7 different distinct actions that you can perform remotely, so my initial take on it was to create
a module for each one so that it can execute its contents via execnet.

But that caused other issues, as you well point out, the remote_exec calls will not share anything.

So I ended up with using a single module for executing all remote functions (as opposed to one module
per each 'actionable') and a couple of helpers for running shell commands via subprocess that would
take a gateway instance to do all the send/receive for me.


> holger
> 
>> 
>> On Mon, Oct 14, 2013 at 1:47 PM, holger krekel <holger at merlinux.eu> wrote:
>> 
>>> On Mon, Oct 14, 2013 at 13:40 -0700, Rick Gerkin wrote:
>>>> Is there anyway to make the contents of the interpreter persist between
>>>> calls to remote_exec?  For example,
>>>> 
>>>> import execnet
>>>> gw = execnet.makegateway("popen//python=jython")
>>>> ch1 = gw.remote_exec('var=2; channel.send(var)')
>>>> ch1.receive() # Prints 2.
>>>> ch2 = gw.remote_exec('channel.send(var)')
>>>> ch2.receive() # NameError: name 'var' is not defined.
>>>> 
>>>> I would expect the gateway to correspond to one continuous python
>>> session,
>>>> but I guess it doesn't?
>>> 
>>> Each remote_exec() has a separate namespace.  There is no execnet-offered
>>> "sharing" space that survives calls.  We could think about something.
>>> Meanwhile you can stick things on an imported module, for example
>>> "sys" or "os" if you don't have own ones.  Or write a little
>>> support code that puts an artifical module into sys.modules,
>>> like this:
>>> 
>>>    def get_shared_data():
>>>        name = "shared_data"
>>>        try:
>>>            return sys.modules[name]
>>>        except KeyError:
>>>            mod = sys.modules[name] = types.ModuleType(name)
>>>            l.append(mod)
>>>            return mod
>>> 
>>> You can then use the result of get_shared_data() to store/retrieve
>>> attributes across sessions.
>>> 
>>> holger
>>> 


From d at tobald.eu.org  Thu Oct 17 11:33:22 2013
From: d at tobald.eu.org (Christophe Siraut)
Date: Thu, 17 Oct 2013 11:33:22 +0200
Subject: [execnet-dev] Calling makegateway with JAVA_OPTIONS
Message-ID: <20131017093321.GB22312@LAP-FHQ-264.fedasil.be>

Hello,

Using execnet.makegateway with jython, it seems I cannot change the java.library.path:

>>> import execnet
>>> gw = execnet.makegateway('popen//python=jython//env:JAVA_OPTIONS=-Djava.library.path=/tmp')
>>> channel = gw.remote_exec("""
...       from java.lang import *
...       channel.send(System.getProperty('java.library.path'))
... """)
>>> print channel.receive()
/usr/lib/jni

While it works from the shell:

$ JAVA_OPTIONS=-Djava.library.path=/tmp jython
Jython 2.5.2 (Debian:hg/91332231a448, May 8 2012, 09:50:46) 
[OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.lang import *
>>> print System.getProperty('java.library.path')
/tmp:/usr/lib/jni

Do I misuse the remote environment variable setting? Do you know any workaround?

Thank you,
Christophe

From holger at merlinux.eu  Mon Oct 21 16:03:42 2013
From: holger at merlinux.eu (holger krekel)
Date: Mon, 21 Oct 2013 14:03:42 +0000
Subject: [execnet-dev] Calling makegateway with JAVA_OPTIONS
In-Reply-To: <20131017093321.GB22312@LAP-FHQ-264.fedasil.be>
References: <20131017093321.GB22312@LAP-FHQ-264.fedasil.be>
Message-ID: <20131021140342.GF3973@merlinux.eu>

Hello Christophe,

On Thu, Oct 17, 2013 at 11:33 +0200, Christophe Siraut wrote:
> Hello,
> 
> Using execnet.makegateway with jython, it seems I cannot change the java.library.path:
> 
> >>> import execnet
> >>> gw = execnet.makegateway('popen//python=jython//env:JAVA_OPTIONS=-Djava.library.path=/tmp')
> >>> channel = gw.remote_exec("""
> ...       from java.lang import *
> ...       channel.send(System.getProperty('java.library.path'))
> ... """)
> >>> print channel.receive()
> /usr/lib/jni
> 
> While it works from the shell:
> 
> $ JAVA_OPTIONS=-Djava.library.path=/tmp jython
> Jython 2.5.2 (Debian:hg/91332231a448, May 8 2012, 09:50:46) 
> [OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from java.lang import *
> >>> print System.getProperty('java.library.path')
> /tmp:/usr/lib/jni
> 
> Do I misuse the remote environment variable setting? Do you know any workaround?

I guess jython needs the env setting at startup already.  But the environment
var is only set after the initial "gateway" bootstrap is sent, i.e. after 
jython started up already.

I guess we could try to set any environment variables when the new subprocess
is spawned (i.e. before jython starts up).  Feel free to file an issue
or, better, prepare a Pull Request on https://bitbucket.org/hpk42/execnet

cheers,
holger

> 
> Thank you,
> Christophe
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at python.org
> https://mail.python.org/mailman/listinfo/execnet-dev
> 

From rick1 at gerkin.org  Tue Oct 22 23:30:47 2013
From: rick1 at gerkin.org (Rick Gerkin)
Date: Tue, 22 Oct 2013 14:30:47 -0700
Subject: [execnet-dev] Calling makegateway with JAVA_OPTIONS
In-Reply-To: <20131021140342.GF3973@merlinux.eu>
References: <20131017093321.GB22312@LAP-FHQ-264.fedasil.be>
 <20131021140342.GF3973@merlinux.eu>
Message-ID: <CAPtMRDW3UQ07YqW-S6WneiOx2E3=aB0teLmA5mA2mcmCQVQi+g@mail.gmail.com>

I wrote a shell script that sets all environment variables, and then runs a
jython socket server script.  I then connect to that socket server using
execnet in python.  I have python launch the shell script via
subprocess.Popen, then check to see that the socket is open, and then
connect via execnet.


On Mon, Oct 21, 2013 at 7:03 AM, holger krekel <holger at merlinux.eu> wrote:

> Hello Christophe,
>
> On Thu, Oct 17, 2013 at 11:33 +0200, Christophe Siraut wrote:
> > Hello,
> >
> > Using execnet.makegateway with jython, it seems I cannot change the
> java.library.path:
> >
> > >>> import execnet
> > >>> gw =
> execnet.makegateway('popen//python=jython//env:JAVA_OPTIONS=-Djava.library.path=/tmp')
> > >>> channel = gw.remote_exec("""
> > ...       from java.lang import *
> > ...       channel.send(System.getProperty('java.library.path'))
> > ... """)
> > >>> print channel.receive()
> > /usr/lib/jni
> >
> > While it works from the shell:
> >
> > $ JAVA_OPTIONS=-Djava.library.path=/tmp jython
> > Jython 2.5.2 (Debian:hg/91332231a448, May 8 2012, 09:50:46)
> > [OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_27
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> from java.lang import *
> > >>> print System.getProperty('java.library.path')
> > /tmp:/usr/lib/jni
> >
> > Do I misuse the remote environment variable setting? Do you know any
> workaround?
>
> I guess jython needs the env setting at startup already.  But the
> environment
> var is only set after the initial "gateway" bootstrap is sent, i.e. after
> jython started up already.
>
> I guess we could try to set any environment variables when the new
> subprocess
> is spawned (i.e. before jython starts up).  Feel free to file an issue
> or, better, prepare a Pull Request on https://bitbucket.org/hpk42/execnet
>
> cheers,
> holger
>
> >
> > Thank you,
> > Christophe
> > _______________________________________________
> > execnet-dev mailing list
> > execnet-dev at python.org
> > https://mail.python.org/mailman/listinfo/execnet-dev
> >
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at python.org
> https://mail.python.org/mailman/listinfo/execnet-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20131022/a0f8d6df/attachment.html>