From charlessolar at gmail.com  Fri Jul  2 18:31:59 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Fri, 2 Jul 2010 11:31:59 -0500
Subject: [execnet-dev] Socket gateway windows service
Message-ID: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com>

I wrote up a windows service script for starting a socket gateway.  Thought
other people might like to use it.  Its pretty basic, but gets the job
done.  PyWin32 is required.

Thanks for the great library.

Charles
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100702/bdc4ea92/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: socketserver.py
Type: text/x-python
Size: 2415 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100702/bdc4ea92/attachment.py>

From charlessolar at gmail.com  Fri Jul  2 23:31:27 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Fri, 2 Jul 2010 16:31:27 -0500
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com>
Message-ID: <AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com>

Ah, a few hours after sending this I found the scripts folder, and here I
thought socketserver.py was just an example.. oh well.
Anyway after using this a bit I am afraid I need to modify how the
Socketgateway works and I would like a tip or two about a couple things.

I noticed that socketgateways will run the remote code in their own world,
which is a bit different from ssh gateways where the ssh session terminates
when the job is done.  The sshd remains untainted because the remote server
spawned a nice python process for us.
However the socketgateways run the remote code in themselves and thus if the
remote code dirties up the interpreter the whole daemon is bad.

In my case I am starting a twisted reactor on the remote server, and once a
twisted reactor has been created its expected that it is alive as long as
the python process itself.  Because of this, the socket gateway daemon is
only good for 1 connection, then it crashes because twisted's reactor has
issues.

The solution to this problem would be to execute the incoming remote code in
a new popen gateway on the remote server instead of inside the socket
gateway instance itself.  I have been skimming a few files but I am not
completely sure how or where would be a good place to put the popen gateway.
My gut tells me the proxy should be put in SocketIO, but I figured it might
save some time to send out an email to find out how these systems interact.

Thanks

On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:

> I wrote up a windows service script for starting a socket gateway.  Thought
> other people might like to use it.  Its pretty basic, but gets the job
> done.  PyWin32 is required.
>
> Thanks for the great library.
>
> Charles
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100702/a97208ae/attachment.html>

From holger at merlinux.eu  Sat Jul  3 10:02:00 2010
From: holger at merlinux.eu (holger krekel)
Date: Sat, 3 Jul 2010 10:02:00 +0200
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com>
	<AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com>
Message-ID: <20100703080159.GD14601@trillke.net>

Hi Charles, 

On Fri, Jul 02, 2010 at 16:31 -0500, Charles Solar wrote:
> Ah, a few hours after sending this I found the scripts folder, and here I
> thought socketserver.py was just an example.. oh well.

Your code still makes sense - an installable windows service would be great. 

> Anyway after using this a bit I am afraid I need to modify how the
> Socketgateway works and I would like a tip or two about a couple things.
> 
> I noticed that socketgateways will run the remote code in their own world,
> which is a bit different from ssh gateways where the ssh session terminates
> when the job is done.  The sshd remains untainted because the remote server
> spawned a nice python process for us.
> However the socketgateways run the remote code in themselves and thus if the
> remote code dirties up the interpreter the whole daemon is bad.

True. 

> In my case I am starting a twisted reactor on the remote server, and once a
> twisted reactor has been created its expected that it is alive as long as
> the python process itself.  Because of this, the socket gateway daemon is
> only good for 1 connection, then it crashes because twisted's reactor has
> issues.
> 
> The solution to this problem would be to execute the incoming remote code in
> a new popen gateway on the remote server instead of inside the socket
> gateway instance itself.  I have been skimming a few files but I am not
> completely sure how or where would be a good place to put the popen gateway.
> My gut tells me the proxy should be put in SocketIO, but I figured it might
> save some time to send out an email to find out how these systems interact.

not sure, but for now i'd rather not try to nest execnet gateways,
mostly because it will be fun to debug (there is some logging and 
nested gateways generally work though).  Also, a nice property of 
the socketgateway server is that it's rather independent from execnet 
impl details.  

Rather, the socketgateway.py service could learn to act just like 
the ssh-daemon by using subprocess.Popen and allowing multiple connections.  
IMHO just using threads that own a socket and proxy IO to their subprocess'ed
gateway could be straightforward.  

In any case, I am happy to review and integrate your code into the next release 
if you go down this route, both for the service and the socketserver issue. 

cheers,
holger


> 
> Thanks
> 
> On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:
> 
> > I wrote up a windows service script for starting a socket gateway.  Thought
> > other people might like to use it.  Its pretty basic, but gets the job
> > done.  PyWin32 is required.
> >
> > Thanks for the great library.
> >
> > Charles
> >

> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev


-- 


From holger at merlinux.eu  Wed Jul  7 15:56:21 2010
From: holger at merlinux.eu (holger krekel)
Date: Wed, 7 Jul 2010 15:56:21 +0200
Subject: [execnet-dev] execnet-1.0.7: new env-setting and keyboardinterrupt
	fixes
Message-ID: <20100707135621.GO14601@trillke.net>


execnet-1.0.7 is a backward compatible release avoiding
a gateway termination race condition and adding the possibility
to add env:NAME=value settings to gateway specifications, 
thanks to thanks Jakub Gustak. 

execnet is a small and stable pure-python library for automatically deploying
and interacting with clusters of Python interpreters.  It seamlessly
instantiates remote interpreters through the 'ssh' command line tool
or socket connections.  It supports interactions between
Python 2.4 through to 3.1, Jython-2.5.1 and pypy-c, therefore enabling
generic Python-to-Python bridging.

execnet-1.0.7 was successfully tested against Python2.7
as well as the usual armada of interpreters ranging 
from Python2.4 to Python3.1, Jython and PyPy. 


See http://codespeak.net/execnet for extensive documentation. 


From charlessolar at gmail.com  Thu Jul  8 17:39:53 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Thu, 8 Jul 2010 10:39:53 -0500
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <20100703080159.GD14601@trillke.net>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com> 
	<AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com> 
	<20100703080159.GD14601@trillke.net>
Message-ID: <AANLkTinsiOYlwwSnSajEzmdbebIt66I3RFpuvQZ6xtex@mail.gmail.com>

Ok I have a solution that works for my purposes.  I rethought my
original idea and figured out how to proxy the io instances to the new
popengateway that socketserver creates, so no modifications to execnet
internals was necessary.
I am attaching my solution, but there are some lines I would like you
to look at and probably fix marked by XXX
For those lines I either used a trick that could break at any update,
or I had to assume things that I am not sure should be assumed.

Look forward to seeing what you do with it :)

Charles

On Sat, Jul 3, 2010 at 3:02 AM, holger krekel <holger at merlinux.eu> wrote:
> Hi Charles,
>
> On Fri, Jul 02, 2010 at 16:31 -0500, Charles Solar wrote:
>> Ah, a few hours after sending this I found the scripts folder, and here I
>> thought socketserver.py was just an example.. oh well.
>
> Your code still makes sense - an installable windows service would be great.
>
>> Anyway after using this a bit I am afraid I need to modify how the
>> Socketgateway works and I would like a tip or two about a couple things.
>>
>> I noticed that socketgateways will run the remote code in their own world,
>> which is a bit different from ssh gateways where the ssh session terminates
>> when the job is done. ?The sshd remains untainted because the remote server
>> spawned a nice python process for us.
>> However the socketgateways run the remote code in themselves and thus if the
>> remote code dirties up the interpreter the whole daemon is bad.
>
> True.
>
>> In my case I am starting a twisted reactor on the remote server, and once a
>> twisted reactor has been created its expected that it is alive as long as
>> the python process itself. ?Because of this, the socket gateway daemon is
>> only good for 1 connection, then it crashes because twisted's reactor has
>> issues.
>>
>> The solution to this problem would be to execute the incoming remote code in
>> a new popen gateway on the remote server instead of inside the socket
>> gateway instance itself. ?I have been skimming a few files but I am not
>> completely sure how or where would be a good place to put the popen gateway.
>> My gut tells me the proxy should be put in SocketIO, but I figured it might
>> save some time to send out an email to find out how these systems interact.
>
> not sure, but for now i'd rather not try to nest execnet gateways,
> mostly because it will be fun to debug (there is some logging and
> nested gateways generally work though). ?Also, a nice property of
> the socketgateway server is that it's rather independent from execnet
> impl details.
>
> Rather, the socketgateway.py service could learn to act just like
> the ssh-daemon by using subprocess.Popen and allowing multiple connections.
> IMHO just using threads that own a socket and proxy IO to their subprocess'ed
> gateway could be straightforward.
>
> In any case, I am happy to review and integrate your code into the next release
> if you go down this route, both for the service and the socketserver issue.
>
> cheers,
> holger
>
>
>>
>> Thanks
>>
>> On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:
>>
>> > I wrote up a windows service script for starting a socket gateway. ?Thought
>> > other people might like to use it. ?Its pretty basic, but gets the job
>> > done. ?PyWin32 is required.
>> >
>> > Thanks for the great library.
>> >
>> > Charles
>> >
>
>> _______________________________________________
>> execnet-dev mailing list
>> execnet-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/execnet-dev
>
>
> --
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: socketserver.py
Type: text/x-python
Size: 4215 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100708/45347c78/attachment.py>

From holger at merlinux.eu  Thu Jul  8 18:33:58 2010
From: holger at merlinux.eu (holger krekel)
Date: Thu, 8 Jul 2010 18:33:58 +0200
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <AANLkTinsiOYlwwSnSajEzmdbebIt66I3RFpuvQZ6xtex@mail.gmail.com>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com>
	<AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com>
	<20100703080159.GD14601@trillke.net>
	<AANLkTinsiOYlwwSnSajEzmdbebIt66I3RFpuvQZ6xtex@mail.gmail.com>
Message-ID: <20100708163358.GR14601@trillke.net>

Hi Charles, 

am quite busy with releasing some bits before i leave for a couple of days
but i am wondering: 

a) could you provide a full example on how to use your code? 

b) why does the IOJoiner need to be aware of (Un)Serializer details? 

cheers,
holger

On Thu, Jul 08, 2010 at 10:39 -0500, Charles Solar wrote:
> Ok I have a solution that works for my purposes.  I rethought my
> original idea and figured out how to proxy the io instances to the new
> popengateway that socketserver creates, so no modifications to execnet
> internals was necessary.
> I am attaching my solution, but there are some lines I would like you
> to look at and probably fix marked by XXX
> For those lines I either used a trick that could break at any update,
> or I had to assume things that I am not sure should be assumed.
> 
> Look forward to seeing what you do with it :)
> 
> Charles
> 
> On Sat, Jul 3, 2010 at 3:02 AM, holger krekel <holger at merlinux.eu> wrote:
> > Hi Charles,
> >
> > On Fri, Jul 02, 2010 at 16:31 -0500, Charles Solar wrote:
> >> Ah, a few hours after sending this I found the scripts folder, and here I
> >> thought socketserver.py was just an example.. oh well.
> >
> > Your code still makes sense - an installable windows service would be great.
> >
> >> Anyway after using this a bit I am afraid I need to modify how the
> >> Socketgateway works and I would like a tip or two about a couple things.
> >>
> >> I noticed that socketgateways will run the remote code in their own world,
> >> which is a bit different from ssh gateways where the ssh session terminates
> >> when the job is done. ?The sshd remains untainted because the remote server
> >> spawned a nice python process for us.
> >> However the socketgateways run the remote code in themselves and thus if the
> >> remote code dirties up the interpreter the whole daemon is bad.
> >
> > True.
> >
> >> In my case I am starting a twisted reactor on the remote server, and once a
> >> twisted reactor has been created its expected that it is alive as long as
> >> the python process itself. ?Because of this, the socket gateway daemon is
> >> only good for 1 connection, then it crashes because twisted's reactor has
> >> issues.
> >>
> >> The solution to this problem would be to execute the incoming remote code in
> >> a new popen gateway on the remote server instead of inside the socket
> >> gateway instance itself. ?I have been skimming a few files but I am not
> >> completely sure how or where would be a good place to put the popen gateway.
> >> My gut tells me the proxy should be put in SocketIO, but I figured it might
> >> save some time to send out an email to find out how these systems interact.
> >
> > not sure, but for now i'd rather not try to nest execnet gateways,
> > mostly because it will be fun to debug (there is some logging and
> > nested gateways generally work though). ?Also, a nice property of
> > the socketgateway server is that it's rather independent from execnet
> > impl details.
> >
> > Rather, the socketgateway.py service could learn to act just like
> > the ssh-daemon by using subprocess.Popen and allowing multiple connections.
> > IMHO just using threads that own a socket and proxy IO to their subprocess'ed
> > gateway could be straightforward.
> >
> > In any case, I am happy to review and integrate your code into the next release
> > if you go down this route, both for the service and the socketserver issue.
> >
> > cheers,
> > holger
> >
> >
> >>
> >> Thanks
> >>
> >> On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:
> >>
> >> > I wrote up a windows service script for starting a socket gateway. ?Thought
> >> > other people might like to use it. ?Its pretty basic, but gets the job
> >> > done. ?PyWin32 is required.
> >> >
> >> > Thanks for the great library.
> >> >
> >> > Charles
> >> >
> >
> >> _______________________________________________
> >> execnet-dev mailing list
> >> execnet-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/execnet-dev
> >
> >
> > --
> >

> """
> Windows service for handling incomming socket gateways
> """
> 
> import threading, SocketServer
> 
> import win32serviceutil
> import win32service
> import win32event
> import servicemanager
> 
> import execnet
> from execnet.gateway_socket import SocketIO
> 
> class IOJoiner():
>     """
>     Joins two io instances so when one wants to read, the incoming data is
>     sent straight to the other io.  Useful for tieing the Popen gateway IO
>     and the SocketIO.
>     """
>     def __init__( self, primaryIO, secondaryIO ):
>         self.prim = primaryIO
>         self.sec = secondaryIO
> 
>     def read( self, numbytes ):
>         buf = self.prim.read( numbytes )
>         self.sec.write( buf )
>         # 'L' corresponds to the 'NoneType' in the Unserializer
>         # Do this so the person who called read does not freak out
>         # XXX I should return an object ACTUALLY representing NoneType, but not sure atm
>         # how to access that data.
>         return 'L'
> 
>     def write( self, data ):
>         # XXX I do not see any reason for the other io to worry
>         # About things the primary is writing, I could be wrong though.
>         self.prim.write( data )
> 
>     def close_read( self ):
>         self.prim.close_read()
> 
>     def close_write( self ):
>         self.prim.close_write()
>             
> 
> class PythonService(win32serviceutil.ServiceFramework):
>     _svc_name_ = "PythonSocketServer"
>     _svc_display_name_ = "Python execnet socket server"
>     _svc_description_ = "Gateway server to allow execnet to connect to windows machines"
>     def __init__(self, args):
>         win32serviceutil.ServiceFramework.__init__(self, args)
>         # Create an event which we will use to wait on.
>         # The "service stop" request will set this event.
>         self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
> 
>     def SvcStop(self):
>         # Before we do anything, tell the SCM we are starting the stop process.
>         self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
>         # And set my event.
>         win32event.SetEvent(self.hWaitStop)
> 
>     def SvcDoRun(self):
>              
>         servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STARTED,(self._svc_name_, ''))
> 
>         self.main()
>         
>         while True:
>             rc = win32event.WaitForSingleObject(self.hWaitStop, 1000)
>             if rc == win32event.WAIT_OBJECT_0:
>                 self._server.shutdown()
>                 self._serverThread.join()
>                 servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STOPPED,(self._svc_name_, ''))
>                 break
> 
>     def main( self ):
> 
>         class TCPHandler( SocketServer.StreamRequestHandler ):
>             def handle( self ):
>                 source = self.rfile.readline().rstrip()
> 
>                 # Spawn a child python process for the new connection
>                 # XXX It would be nice if there was a way to remotely request a new ironpython or jython instance instead of a generic cpython instance
>                 # but that would require modifications to SocketGateway I believe.
>                 gw = execnet.PopenGateway(python='python')
> 
>                 sockio = SocketIO( self.request )
>                 gwio = gw._io
>                 
>                 source = source.replace( "io = SocketIO(clientsock)", "io = joiner" )
>                 
>                 g = { 'joiner': IOJoiner( sockio, gwio ), 'address': self.client_address }
>                 gw._io = IOJoiner( gwio, sockio )
> 
>                 source = eval(source)
>                 
>                 if source:
>                     co = compile( source+'\n', source, 'exec' )
> 
>                     try:
>                         exec co in g
>                     except Exception as e:
>                         servicemanager.LogErrorMsg( "Execution of received source code raised the following exception: %r" % e )
>                 
>         self._server = SocketServer.ThreadingTCPServer( ('0.0.0.0', 8888), TCPHandler )
>         self._serverThread = threading.Thread( target=self._server.serve_forever )
>         self._serverThread.setDaemon( True )
>         self._serverThread.start()
> 
> if __name__ == '__main__':
>     win32serviceutil.HandleCommandLine(PythonService)


-- 


From charlessolar at gmail.com  Thu Jul  8 19:03:27 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Thu, 8 Jul 2010 12:03:27 -0500
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <20100708163358.GR14601@trillke.net>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com> 
	<AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com> 
	<20100703080159.GD14601@trillke.net>
	<AANLkTinsiOYlwwSnSajEzmdbebIt66I3RFpuvQZ6xtex@mail.gmail.com> 
	<20100708163358.GR14601@trillke.net>
Message-ID: <AANLkTimV-bJJq_-53oR2KUt92eDNmcmYVAdZRvhylqEC@mail.gmail.com>

Well its a windows service so on any windows machine you would just do

python socketserver.py install
python socketserver.py start

to get the socketserver running on port 8888.

You can connect to it from any machine with
import execnet
gw=execnet.makegateway( "socket=[hostname]:8888" )

on the socket server it will spawn a new popen gateway and connect the
socketIO to the popenIO through IOJoiner.

IOJoiner replaces the io that the gateways on the server side use to
serve requests.  So when the slave socket gateway calls io.read(1) in
the serve loop, its actually calling IOJoiner.read().  IOJoiner.read()
will then read from the socket, and write all incoming data to the
popen gateway.
However, the socket slave gateway still wants a return value.
Returning what we just read from the socket will produce incorrect
results, so I have it just return the Unserializer's None type, which
will cause the serve loop to not do anything, exactly what I wanted.
On the popen end, I write the incoming socket data to the master
gateway's io, which is then read by the slave gateway and used.  Also,
all data read from the popen slave gateway is written to the socket
io, same as how socket io passes to popen io.

IOJoiner needed to use Unserializer types because I did not want the
slave socket gateway or the master popen io do anything, as their only
purpose is to facilitate communication between the other io gateways.

I have no doubt there is a better way to do this, but with my limited
understanding of execnet internals at the moment, this seems like the
easiest solution.

Charles

On Thu, Jul 8, 2010 at 11:33 AM, holger krekel <holger at merlinux.eu> wrote:
> Hi Charles,
>
> am quite busy with releasing some bits before i leave for a couple of days
> but i am wondering:
>
> a) could you provide a full example on how to use your code?
>
> b) why does the IOJoiner need to be aware of (Un)Serializer details?
>
> cheers,
> holger
>
> On Thu, Jul 08, 2010 at 10:39 -0500, Charles Solar wrote:
>> Ok I have a solution that works for my purposes. ?I rethought my
>> original idea and figured out how to proxy the io instances to the new
>> popengateway that socketserver creates, so no modifications to execnet
>> internals was necessary.
>> I am attaching my solution, but there are some lines I would like you
>> to look at and probably fix marked by XXX
>> For those lines I either used a trick that could break at any update,
>> or I had to assume things that I am not sure should be assumed.
>>
>> Look forward to seeing what you do with it :)
>>
>> Charles
>>
>> On Sat, Jul 3, 2010 at 3:02 AM, holger krekel <holger at merlinux.eu> wrote:
>> > Hi Charles,
>> >
>> > On Fri, Jul 02, 2010 at 16:31 -0500, Charles Solar wrote:
>> >> Ah, a few hours after sending this I found the scripts folder, and here I
>> >> thought socketserver.py was just an example.. oh well.
>> >
>> > Your code still makes sense - an installable windows service would be great.
>> >
>> >> Anyway after using this a bit I am afraid I need to modify how the
>> >> Socketgateway works and I would like a tip or two about a couple things.
>> >>
>> >> I noticed that socketgateways will run the remote code in their own world,
>> >> which is a bit different from ssh gateways where the ssh session terminates
>> >> when the job is done. ?The sshd remains untainted because the remote server
>> >> spawned a nice python process for us.
>> >> However the socketgateways run the remote code in themselves and thus if the
>> >> remote code dirties up the interpreter the whole daemon is bad.
>> >
>> > True.
>> >
>> >> In my case I am starting a twisted reactor on the remote server, and once a
>> >> twisted reactor has been created its expected that it is alive as long as
>> >> the python process itself. ?Because of this, the socket gateway daemon is
>> >> only good for 1 connection, then it crashes because twisted's reactor has
>> >> issues.
>> >>
>> >> The solution to this problem would be to execute the incoming remote code in
>> >> a new popen gateway on the remote server instead of inside the socket
>> >> gateway instance itself. ?I have been skimming a few files but I am not
>> >> completely sure how or where would be a good place to put the popen gateway.
>> >> My gut tells me the proxy should be put in SocketIO, but I figured it might
>> >> save some time to send out an email to find out how these systems interact.
>> >
>> > not sure, but for now i'd rather not try to nest execnet gateways,
>> > mostly because it will be fun to debug (there is some logging and
>> > nested gateways generally work though). ?Also, a nice property of
>> > the socketgateway server is that it's rather independent from execnet
>> > impl details.
>> >
>> > Rather, the socketgateway.py service could learn to act just like
>> > the ssh-daemon by using subprocess.Popen and allowing multiple connections.
>> > IMHO just using threads that own a socket and proxy IO to their subprocess'ed
>> > gateway could be straightforward.
>> >
>> > In any case, I am happy to review and integrate your code into the next release
>> > if you go down this route, both for the service and the socketserver issue.
>> >
>> > cheers,
>> > holger
>> >
>> >
>> >>
>> >> Thanks
>> >>
>> >> On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:
>> >>
>> >> > I wrote up a windows service script for starting a socket gateway. ?Thought
>> >> > other people might like to use it. ?Its pretty basic, but gets the job
>> >> > done. ?PyWin32 is required.
>> >> >
>> >> > Thanks for the great library.
>> >> >
>> >> > Charles
>> >> >
>> >
>> >> _______________________________________________
>> >> execnet-dev mailing list
>> >> execnet-dev at codespeak.net
>> >> http://codespeak.net/mailman/listinfo/execnet-dev
>> >
>> >
>> > --
>> >
>
>> """
>> Windows service for handling incomming socket gateways
>> """
>>
>> import threading, SocketServer
>>
>> import win32serviceutil
>> import win32service
>> import win32event
>> import servicemanager
>>
>> import execnet
>> from execnet.gateway_socket import SocketIO
>>
>> class IOJoiner():
>> ? ? """
>> ? ? Joins two io instances so when one wants to read, the incoming data is
>> ? ? sent straight to the other io. ?Useful for tieing the Popen gateway IO
>> ? ? and the SocketIO.
>> ? ? """
>> ? ? def __init__( self, primaryIO, secondaryIO ):
>> ? ? ? ? self.prim = primaryIO
>> ? ? ? ? self.sec = secondaryIO
>>
>> ? ? def read( self, numbytes ):
>> ? ? ? ? buf = self.prim.read( numbytes )
>> ? ? ? ? self.sec.write( buf )
>> ? ? ? ? # 'L' corresponds to the 'NoneType' in the Unserializer
>> ? ? ? ? # Do this so the person who called read does not freak out
>> ? ? ? ? # XXX I should return an object ACTUALLY representing NoneType, but not sure atm
>> ? ? ? ? # how to access that data.
>> ? ? ? ? return 'L'
>>
>> ? ? def write( self, data ):
>> ? ? ? ? # XXX I do not see any reason for the other io to worry
>> ? ? ? ? # About things the primary is writing, I could be wrong though.
>> ? ? ? ? self.prim.write( data )
>>
>> ? ? def close_read( self ):
>> ? ? ? ? self.prim.close_read()
>>
>> ? ? def close_write( self ):
>> ? ? ? ? self.prim.close_write()
>>
>>
>> class PythonService(win32serviceutil.ServiceFramework):
>> ? ? _svc_name_ = "PythonSocketServer"
>> ? ? _svc_display_name_ = "Python execnet socket server"
>> ? ? _svc_description_ = "Gateway server to allow execnet to connect to windows machines"
>> ? ? def __init__(self, args):
>> ? ? ? ? win32serviceutil.ServiceFramework.__init__(self, args)
>> ? ? ? ? # Create an event which we will use to wait on.
>> ? ? ? ? # The "service stop" request will set this event.
>> ? ? ? ? self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
>>
>> ? ? def SvcStop(self):
>> ? ? ? ? # Before we do anything, tell the SCM we are starting the stop process.
>> ? ? ? ? self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
>> ? ? ? ? # And set my event.
>> ? ? ? ? win32event.SetEvent(self.hWaitStop)
>>
>> ? ? def SvcDoRun(self):
>>
>> ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STARTED,(self._svc_name_, ''))
>>
>> ? ? ? ? self.main()
>>
>> ? ? ? ? while True:
>> ? ? ? ? ? ? rc = win32event.WaitForSingleObject(self.hWaitStop, 1000)
>> ? ? ? ? ? ? if rc == win32event.WAIT_OBJECT_0:
>> ? ? ? ? ? ? ? ? self._server.shutdown()
>> ? ? ? ? ? ? ? ? self._serverThread.join()
>> ? ? ? ? ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STOPPED,(self._svc_name_, ''))
>> ? ? ? ? ? ? ? ? break
>>
>> ? ? def main( self ):
>>
>> ? ? ? ? class TCPHandler( SocketServer.StreamRequestHandler ):
>> ? ? ? ? ? ? def handle( self ):
>> ? ? ? ? ? ? ? ? source = self.rfile.readline().rstrip()
>>
>> ? ? ? ? ? ? ? ? # Spawn a child python process for the new connection
>> ? ? ? ? ? ? ? ? # XXX It would be nice if there was a way to remotely request a new ironpython or jython instance instead of a generic cpython instance
>> ? ? ? ? ? ? ? ? # but that would require modifications to SocketGateway I believe.
>> ? ? ? ? ? ? ? ? gw = execnet.PopenGateway(python='python')
>>
>> ? ? ? ? ? ? ? ? sockio = SocketIO( self.request )
>> ? ? ? ? ? ? ? ? gwio = gw._io
>>
>> ? ? ? ? ? ? ? ? source = source.replace( "io = SocketIO(clientsock)", "io = joiner" )
>>
>> ? ? ? ? ? ? ? ? g = { 'joiner': IOJoiner( sockio, gwio ), 'address': self.client_address }
>> ? ? ? ? ? ? ? ? gw._io = IOJoiner( gwio, sockio )
>>
>> ? ? ? ? ? ? ? ? source = eval(source)
>>
>> ? ? ? ? ? ? ? ? if source:
>> ? ? ? ? ? ? ? ? ? ? co = compile( source+'\n', source, 'exec' )
>>
>> ? ? ? ? ? ? ? ? ? ? try:
>> ? ? ? ? ? ? ? ? ? ? ? ? exec co in g
>> ? ? ? ? ? ? ? ? ? ? except Exception as e:
>> ? ? ? ? ? ? ? ? ? ? ? ? servicemanager.LogErrorMsg( "Execution of received source code raised the following exception: %r" % e )
>>
>> ? ? ? ? self._server = SocketServer.ThreadingTCPServer( ('0.0.0.0', 8888), TCPHandler )
>> ? ? ? ? self._serverThread = threading.Thread( target=self._server.serve_forever )
>> ? ? ? ? self._serverThread.setDaemon( True )
>> ? ? ? ? self._serverThread.start()
>>
>> if __name__ == '__main__':
>> ? ? win32serviceutil.HandleCommandLine(PythonService)
>
>
> --
>


From charlessolar at gmail.com  Fri Jul  9 17:03:38 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Fri, 9 Jul 2010 10:03:38 -0500
Subject: [execnet-dev] RSync Issue
Message-ID: <AANLkTilS-Xxx1uxslsFvb0Wpu9bOJZjm-dFtGpPHkFJm@mail.gmail.com>

It would appear that RSync will only work with brand new gateways that
have not had any other operations run on them.  Is this intentional?
For example, I do something like

gw = execnet.makegateway( "ssh=foobar" )
gw.remote_exec( mybootstrapper )

... then later I want to rsync ...

rsync = execnet.RSync( "/tmp/foobar" )
rsync.add_target( gw, "/tmp/barfoo" )
rsync.send()

I get remote EOFErrors

File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
    self._end_of_channel(channel)
  File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
    channel.waitclose()
  File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
    raise error
exceptions.EOFError: expected 1 bytes, got 0

If I however create a brand new gateway to give to rsync it works fine.

I would think that since rsync just creates a new channel on the
gateway that it should work on existing 'old' gateways.

Charles


From holger at merlinux.eu  Sun Jul 11 12:36:18 2010
From: holger at merlinux.eu (holger krekel)
Date: Sun, 11 Jul 2010 12:36:18 +0200
Subject: [execnet-dev] RSync Issue
In-Reply-To: <AANLkTilS-Xxx1uxslsFvb0Wpu9bOJZjm-dFtGpPHkFJm@mail.gmail.com>
References: <AANLkTilS-Xxx1uxslsFvb0Wpu9bOJZjm-dFtGpPHkFJm@mail.gmail.com>
Message-ID: <20100711103618.GT14601@trillke.net>

Hi Charles, 

On Fri, Jul 09, 2010 at 10:03 -0500, Charles Solar wrote:
> It would appear that RSync will only work with brand new gateways that
> have not had any other operations run on them.  Is this intentional?

No.  Sounds like a bug. 

> For example, I do something like
> 
> gw = execnet.makegateway( "ssh=foobar" )
> gw.remote_exec( mybootstrapper )
> 
> ... then later I want to rsync ...
> 
> rsync = execnet.RSync( "/tmp/foobar" )
> rsync.add_target( gw, "/tmp/barfoo" )
> rsync.send()
> 
> I get remote EOFErrors
> 
> File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
>     self._end_of_channel(channel)
>   File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
>     channel.waitclose()
>   File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
>     raise error
> exceptions.EOFError: expected 1 bytes, got 0
> 
> If I however create a brand new gateway to give to rsync it works fine.
> 
> I would think that since rsync just creates a new channel on the
> gateway that it should work on existing 'old' gateways.

yip. Myself i only used it for doing things at the beginning, it seems.
Do you have or could you try to find an repeatable minimal example? 

best,

holger


From holger at merlinux.eu  Sun Jul 11 14:01:02 2010
From: holger at merlinux.eu (holger krekel)
Date: Sun, 11 Jul 2010 14:01:02 +0200
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <AANLkTimV-bJJq_-53oR2KUt92eDNmcmYVAdZRvhylqEC@mail.gmail.com>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com>
	<AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com>
	<20100703080159.GD14601@trillke.net>
	<AANLkTinsiOYlwwSnSajEzmdbebIt66I3RFpuvQZ6xtex@mail.gmail.com>
	<20100708163358.GR14601@trillke.net>
	<AANLkTimV-bJJq_-53oR2KUt92eDNmcmYVAdZRvhylqEC@mail.gmail.com>
Message-ID: <20100711120102.GV14601@trillke.net>

Hi Charles, 

On Thu, Jul 08, 2010 at 12:03 -0500, Charles Solar wrote:
> Well its a windows service so on any windows machine you would just do
> 
> python socketserver.py install
> python socketserver.py start
> 
> to get the socketserver running on port 8888.

i've never seen or done this, actually.  Thanks :)

> You can connect to it from any machine with
> import execnet
> gw=execnet.makegateway( "socket=[hostname]:8888" )
> 
> on the socket server it will spawn a new popen gateway and connect the
> socketIO to the popenIO through IOJoiner.
> 
> IOJoiner replaces the io that the gateways on the server side use to
> serve requests.  So when the slave socket gateway calls io.read(1) in
> the serve loop, its actually calling IOJoiner.read().  IOJoiner.read()
> will then read from the socket, and write all incoming data to the
> popen gateway.

Let me introduce some terms for the three processes in question: 

    Invocation         Mediator      Worker 
              <socket>          <popen> 

And then try to rephrase what you said to see if i understood: 
The Mediator and Invocation process use a SocketGateway
and the Worker and Mediator a PopenGateway.  The Mediator
replaces both the Mediator-Invocation IO and and the Mediator-Worker
IO with Joiner instances.  

Then you run the SocketGateway serve code which reads from the
Mediator-Invocation Joiner-IO and forward the raw data 
to the *original* Worker-IO.  The Receiver-Thread of the Mediator-Worker
PopenGateway also uses the joiner and what it reads it sends
to the original Invocation IO. 

Requires some concentration to think about this :) 

This method you are using requires that the socketservice side
has execnet installed and is a bit fragile wrt to execnet 
integration (the "L" bit although we could maybe introduce a 
special exception for that). Cool that it works :) 

I am not sure yet how an alternative could look like. 
Here are my current thoughts helping also myself
to get clearer. 

I'd like to have something higher level, maybe introduce
internally the concept of an "Instantiator", that would
get called to "open a subprocess with a bootstrap" and
return an IO object.  It would be naturally used by 
popen-style gateways. This internal refactoring is 
just an increment how things are now. 

We could then extend the instantiator to make it 
call "subprocess.Popen" not directly but rather
through an existing Gateway, returning an IO 
object on the invocation side that is based
on communication with the mediator through the/a channel. 

For bootstrapping a new gateway the Invocation/instantiator
would send an "open a subprocess with bootstrap" command
to a loop which it installed on the Mediator. 
The Mediator would call "subprocess.Popen" with the
given bootstrap and then read/write from the subprocess
pipes and read/write to the channel which connects
the Invocation and Mediator.  

If i see it correctly the Mediator would basically 
do popen.stdin.read(n) and if it reads something 
write it to the Invocation/Mediator channel. 
And if the Mediator receives data a channel callback 
(usually running in the receiver-thread) 
could send it to Popen.write(s). 

This way execnet would not need to be installed 
at the Mediator side and the concept should work
uniformly for all gateway types ... 

cheers,
holger

> However, the socket slave gateway still wants a return value.
>
> Returning what we just read from the socket will produce incorrect
> results, so I have it just return the Unserializer's None type, which
> will cause the serve loop to not do anything, exactly what I wanted.
>
> On the popen end, I write the incoming socket data to the master
> gateway's io, which is then read by the slave gateway and used.  Also,
> all data read from the popen slave gateway is written to the socket
> io, same as how socket io passes to popen io.
> 
> IOJoiner needed to use Unserializer types because I did not want the
> slave socket gateway or the master popen io do anything, as their only
> purpose is to facilitate communication between the other io gateways.
> 
> I have no doubt there is a better way to do this, but with my limited
> understanding of execnet internals at the moment, this seems like the
> easiest solution.
> 
> Charles
> 
> On Thu, Jul 8, 2010 at 11:33 AM, holger krekel <holger at merlinux.eu> wrote:
> > Hi Charles,
> >
> > am quite busy with releasing some bits before i leave for a couple of days
> > but i am wondering:
> >
> > a) could you provide a full example on how to use your code?
> >
> > b) why does the IOJoiner need to be aware of (Un)Serializer details?
> >
> > cheers,
> > holger
> >
> > On Thu, Jul 08, 2010 at 10:39 -0500, Charles Solar wrote:
> >> Ok I have a solution that works for my purposes. ?I rethought my
> >> original idea and figured out how to proxy the io instances to the new
> >> popengateway that socketserver creates, so no modifications to execnet
> >> internals was necessary.
> >> I am attaching my solution, but there are some lines I would like you
> >> to look at and probably fix marked by XXX
> >> For those lines I either used a trick that could break at any update,
> >> or I had to assume things that I am not sure should be assumed.
> >>
> >> Look forward to seeing what you do with it :)
> >>
> >> Charles
> >>
> >> On Sat, Jul 3, 2010 at 3:02 AM, holger krekel <holger at merlinux.eu> wrote:
> >> > Hi Charles,
> >> >
> >> > On Fri, Jul 02, 2010 at 16:31 -0500, Charles Solar wrote:
> >> >> Ah, a few hours after sending this I found the scripts folder, and here I
> >> >> thought socketserver.py was just an example.. oh well.
> >> >
> >> > Your code still makes sense - an installable windows service would be great.
> >> >
> >> >> Anyway after using this a bit I am afraid I need to modify how the
> >> >> Socketgateway works and I would like a tip or two about a couple things.
> >> >>
> >> >> I noticed that socketgateways will run the remote code in their own world,
> >> >> which is a bit different from ssh gateways where the ssh session terminates
> >> >> when the job is done. ?The sshd remains untainted because the remote server
> >> >> spawned a nice python process for us.
> >> >> However the socketgateways run the remote code in themselves and thus if the
> >> >> remote code dirties up the interpreter the whole daemon is bad.
> >> >
> >> > True.
> >> >
> >> >> In my case I am starting a twisted reactor on the remote server, and once a
> >> >> twisted reactor has been created its expected that it is alive as long as
> >> >> the python process itself. ?Because of this, the socket gateway daemon is
> >> >> only good for 1 connection, then it crashes because twisted's reactor has
> >> >> issues.
> >> >>
> >> >> The solution to this problem would be to execute the incoming remote code in
> >> >> a new popen gateway on the remote server instead of inside the socket
> >> >> gateway instance itself. ?I have been skimming a few files but I am not
> >> >> completely sure how or where would be a good place to put the popen gateway.
> >> >> My gut tells me the proxy should be put in SocketIO, but I figured it might
> >> >> save some time to send out an email to find out how these systems interact.
> >> >
> >> > not sure, but for now i'd rather not try to nest execnet gateways,
> >> > mostly because it will be fun to debug (there is some logging and
> >> > nested gateways generally work though). ?Also, a nice property of
> >> > the socketgateway server is that it's rather independent from execnet
> >> > impl details.
> >> >
> >> > Rather, the socketgateway.py service could learn to act just like
> >> > the ssh-daemon by using subprocess.Popen and allowing multiple connections.
> >> > IMHO just using threads that own a socket and proxy IO to their subprocess'ed
> >> > gateway could be straightforward.
> >> >
> >> > In any case, I am happy to review and integrate your code into the next release
> >> > if you go down this route, both for the service and the socketserver issue.
> >> >
> >> > cheers,
> >> > holger
> >> >
> >> >
> >> >>
> >> >> Thanks
> >> >>
> >> >> On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:
> >> >>
> >> >> > I wrote up a windows service script for starting a socket gateway. ?Thought
> >> >> > other people might like to use it. ?Its pretty basic, but gets the job
> >> >> > done. ?PyWin32 is required.
> >> >> >
> >> >> > Thanks for the great library.
> >> >> >
> >> >> > Charles
> >> >> >
> >> >
> >> >> _______________________________________________
> >> >> execnet-dev mailing list
> >> >> execnet-dev at codespeak.net
> >> >> http://codespeak.net/mailman/listinfo/execnet-dev
> >> >
> >> >
> >> > --
> >> >
> >
> >> """
> >> Windows service for handling incomming socket gateways
> >> """
> >>
> >> import threading, SocketServer
> >>
> >> import win32serviceutil
> >> import win32service
> >> import win32event
> >> import servicemanager
> >>
> >> import execnet
> >> from execnet.gateway_socket import SocketIO
> >>
> >> class IOJoiner():
> >> ? ? """
> >> ? ? Joins two io instances so when one wants to read, the incoming data is
> >> ? ? sent straight to the other io. ?Useful for tieing the Popen gateway IO
> >> ? ? and the SocketIO.
> >> ? ? """
> >> ? ? def __init__( self, primaryIO, secondaryIO ):
> >> ? ? ? ? self.prim = primaryIO
> >> ? ? ? ? self.sec = secondaryIO
> >>
> >> ? ? def read( self, numbytes ):
> >> ? ? ? ? buf = self.prim.read( numbytes )
> >> ? ? ? ? self.sec.write( buf )
> >> ? ? ? ? # 'L' corresponds to the 'NoneType' in the Unserializer
> >> ? ? ? ? # Do this so the person who called read does not freak out
> >> ? ? ? ? # XXX I should return an object ACTUALLY representing NoneType, but not sure atm
> >> ? ? ? ? # how to access that data.
> >> ? ? ? ? return 'L'
> >>
> >> ? ? def write( self, data ):
> >> ? ? ? ? # XXX I do not see any reason for the other io to worry
> >> ? ? ? ? # About things the primary is writing, I could be wrong though.
> >> ? ? ? ? self.prim.write( data )
> >>
> >> ? ? def close_read( self ):
> >> ? ? ? ? self.prim.close_read()
> >>
> >> ? ? def close_write( self ):
> >> ? ? ? ? self.prim.close_write()
> >>
> >>
> >> class PythonService(win32serviceutil.ServiceFramework):
> >> ? ? _svc_name_ = "PythonSocketServer"
> >> ? ? _svc_display_name_ = "Python execnet socket server"
> >> ? ? _svc_description_ = "Gateway server to allow execnet to connect to windows machines"
> >> ? ? def __init__(self, args):
> >> ? ? ? ? win32serviceutil.ServiceFramework.__init__(self, args)
> >> ? ? ? ? # Create an event which we will use to wait on.
> >> ? ? ? ? # The "service stop" request will set this event.
> >> ? ? ? ? self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
> >>
> >> ? ? def SvcStop(self):
> >> ? ? ? ? # Before we do anything, tell the SCM we are starting the stop process.
> >> ? ? ? ? self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
> >> ? ? ? ? # And set my event.
> >> ? ? ? ? win32event.SetEvent(self.hWaitStop)
> >>
> >> ? ? def SvcDoRun(self):
> >>
> >> ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STARTED,(self._svc_name_, ''))
> >>
> >> ? ? ? ? self.main()
> >>
> >> ? ? ? ? while True:
> >> ? ? ? ? ? ? rc = win32event.WaitForSingleObject(self.hWaitStop, 1000)
> >> ? ? ? ? ? ? if rc == win32event.WAIT_OBJECT_0:
> >> ? ? ? ? ? ? ? ? self._server.shutdown()
> >> ? ? ? ? ? ? ? ? self._serverThread.join()
> >> ? ? ? ? ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STOPPED,(self._svc_name_, ''))
> >> ? ? ? ? ? ? ? ? break
> >>
> >> ? ? def main( self ):
> >>
> >> ? ? ? ? class TCPHandler( SocketServer.StreamRequestHandler ):
> >> ? ? ? ? ? ? def handle( self ):
> >> ? ? ? ? ? ? ? ? source = self.rfile.readline().rstrip()
> >>
> >> ? ? ? ? ? ? ? ? # Spawn a child python process for the new connection
> >> ? ? ? ? ? ? ? ? # XXX It would be nice if there was a way to remotely request a new ironpython or jython instance instead of a generic cpython instance
> >> ? ? ? ? ? ? ? ? # but that would require modifications to SocketGateway I believe.
> >> ? ? ? ? ? ? ? ? gw = execnet.PopenGateway(python='python')
> >>
> >> ? ? ? ? ? ? ? ? sockio = SocketIO( self.request )
> >> ? ? ? ? ? ? ? ? gwio = gw._io
> >>
> >> ? ? ? ? ? ? ? ? source = source.replace( "io = SocketIO(clientsock)", "io = joiner" )
> >>
> >> ? ? ? ? ? ? ? ? g = { 'joiner': IOJoiner( sockio, gwio ), 'address': self.client_address }
> >> ? ? ? ? ? ? ? ? gw._io = IOJoiner( gwio, sockio )
> >>
> >> ? ? ? ? ? ? ? ? source = eval(source)
> >>
> >> ? ? ? ? ? ? ? ? if source:
> >> ? ? ? ? ? ? ? ? ? ? co = compile( source+'\n', source, 'exec' )
> >>
> >> ? ? ? ? ? ? ? ? ? ? try:
> >> ? ? ? ? ? ? ? ? ? ? ? ? exec co in g
> >> ? ? ? ? ? ? ? ? ? ? except Exception as e:
> >> ? ? ? ? ? ? ? ? ? ? ? ? servicemanager.LogErrorMsg( "Execution of received source code raised the following exception: %r" % e )
> >>
> >> ? ? ? ? self._server = SocketServer.ThreadingTCPServer( ('0.0.0.0', 8888), TCPHandler )
> >> ? ? ? ? self._serverThread = threading.Thread( target=self._server.serve_forever )
> >> ? ? ? ? self._serverThread.setDaemon( True )
> >> ? ? ? ? self._serverThread.start()
> >>
> >> if __name__ == '__main__':
> >> ? ? win32serviceutil.HandleCommandLine(PythonService)
> >
> >
> > --
> >
> 

-- 


From charlessolar at gmail.com  Sun Jul 11 19:01:00 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Sun, 11 Jul 2010 12:01:00 -0500
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <20100711120102.GV14601@trillke.net>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com> 
	<AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com> 
	<20100703080159.GD14601@trillke.net>
	<AANLkTinsiOYlwwSnSajEzmdbebIt66I3RFpuvQZ6xtex@mail.gmail.com> 
	<20100708163358.GR14601@trillke.net>
	<AANLkTimV-bJJq_-53oR2KUt92eDNmcmYVAdZRvhylqEC@mail.gmail.com> 
	<20100711120102.GV14601@trillke.net>
Message-ID: <AANLkTim7qXuJg2FjvEq5k5C6gbY1Pg8HVl87YzxTgyV2@mail.gmail.com>

Comments inline for this one :)

On Sun, Jul 11, 2010 at 7:01 AM, holger krekel <holger at merlinux.eu> wrote:
> Hi Charles,
>
> On Thu, Jul 08, 2010 at 12:03 -0500, Charles Solar wrote:
>> Well its a windows service so on any windows machine you would just do
>>
>> python socketserver.py install
>> python socketserver.py start
>>
>> to get the socketserver running on port 8888.
>
> i've never seen or done this, actually. ?Thanks :)
>
>> You can connect to it from any machine with
>> import execnet
>> gw=execnet.makegateway( "socket=[hostname]:8888" )
>>
>> on the socket server it will spawn a new popen gateway and connect the
>> socketIO to the popenIO through IOJoiner.
>>
>> IOJoiner replaces the io that the gateways on the server side use to
>> serve requests. ?So when the slave socket gateway calls io.read(1) in
>> the serve loop, its actually calling IOJoiner.read(). ?IOJoiner.read()
>> will then read from the socket, and write all incoming data to the
>> popen gateway.
>
> Let me introduce some terms for the three processes in question:
>
> ? ?Invocation ? ? ? ? Mediator ? ? ?Worker
> ? ? ? ? ? ? ?<socket> ? ? ? ? ?<popen>
>
> And then try to rephrase what you said to see if i understood:
> The Mediator and Invocation process use a SocketGateway
> and the Worker and Mediator a PopenGateway. ?The Mediator
> replaces both the Mediator-Invocation IO and and the Mediator-Worker
> IO with Joiner instances.
>
> Then you run the SocketGateway serve code which reads from the
> Mediator-Invocation Joiner-IO and forward the raw data
> to the *original* Worker-IO. ?The Receiver-Thread of the Mediator-Worker
> PopenGateway also uses the joiner and what it reads it sends
> to the original Invocation IO.

yep, absolutely correct.

>
> Requires some concentration to think about this :)

took me several hours of meditation to come up with it haha.

>
> This method you are using requires that the socketservice side
> has execnet installed and is a bit fragile wrt to execnet
> integration (the "L" bit although we could maybe introduce a
> special exception for that). Cool that it works :)
>
> I am not sure yet how an alternative could look like.
> Here are my current thoughts helping also myself
> to get clearer.
>
> I'd like to have something higher level, maybe introduce
> internally the concept of an "Instantiator", that would
> get called to "open a subprocess with a bootstrap" and
> return an IO object. ?It would be naturally used by
> popen-style gateways. This internal refactoring is
> just an increment how things are now.
>
> We could then extend the instantiator to make it
> call "subprocess.Popen" not directly but rather
> through an existing Gateway, returning an IO
> object on the invocation side that is based
> on communication with the mediator through the/a channel.
>
> For bootstrapping a new gateway the Invocation/instantiator
> would send an "open a subprocess with bootstrap" command
> to a loop which it installed on the Mediator.
> The Mediator would call "subprocess.Popen" with the
> given bootstrap and then read/write from the subprocess
> pipes and read/write to the channel which connects
> the Invocation and Mediator.

Yea that is where I was stuck because I could not modify what the
invoker to send a different bootstrap to the socket server.  I
actually had to do a string replace with the given bootstrap code to
make it use the joiner class.  Which I really did not like doing.

>
> If i see it correctly the Mediator would basically
> do popen.stdin.read(n) and if it reads something
> write it to the Invocation/Mediator channel.
> And if the Mediator receives data a channel callback
> (usually running in the receiver-thread)
> could send it to Popen.write(s).
>
> This way execnet would not need to be installed
> at the Mediator side and the concept should work
> uniformly for all gateway types ...
>
> cheers,
> holger
>

Sounds good but I have to go digging around in the code more to
understand where things are connected more clearly.  I was limited by
not wanting to change execnet internally but if you think this can be
done with a minimal change to add new features to the popen gateway
then that sounds like the best route.
There is something I found out while using my solution the other day.
I was using RSync to a machine using this socket service and the file
transfers were taking minutes to send files of 1-2 megs in size.  It
appears this io mediation takes its toll on communication speed.
Because for rsync its not necessary to spawn a child process and
communicate through pipes, I decided to open a new socket that acts
like the original socket server and executes the commands without
spawning a child process.  Which fixes the rsync issue.
I mention it because while we are refactoring this io a bit it might
be a good idea to make a signal for 'dont spawn child process' or
something.

Charles

>> However, the socket slave gateway still wants a return value.
>>
>> Returning what we just read from the socket will produce incorrect
>> results, so I have it just return the Unserializer's None type, which
>> will cause the serve loop to not do anything, exactly what I wanted.
>>
>> On the popen end, I write the incoming socket data to the master
>> gateway's io, which is then read by the slave gateway and used. ?Also,
>> all data read from the popen slave gateway is written to the socket
>> io, same as how socket io passes to popen io.
>>
>> IOJoiner needed to use Unserializer types because I did not want the
>> slave socket gateway or the master popen io do anything, as their only
>> purpose is to facilitate communication between the other io gateways.
>>
>> I have no doubt there is a better way to do this, but with my limited
>> understanding of execnet internals at the moment, this seems like the
>> easiest solution.
>>
>> Charles
>>
>> On Thu, Jul 8, 2010 at 11:33 AM, holger krekel <holger at merlinux.eu> wrote:
>> > Hi Charles,
>> >
>> > am quite busy with releasing some bits before i leave for a couple of days
>> > but i am wondering:
>> >
>> > a) could you provide a full example on how to use your code?
>> >
>> > b) why does the IOJoiner need to be aware of (Un)Serializer details?
>> >
>> > cheers,
>> > holger
>> >
>> > On Thu, Jul 08, 2010 at 10:39 -0500, Charles Solar wrote:
>> >> Ok I have a solution that works for my purposes. ?I rethought my
>> >> original idea and figured out how to proxy the io instances to the new
>> >> popengateway that socketserver creates, so no modifications to execnet
>> >> internals was necessary.
>> >> I am attaching my solution, but there are some lines I would like you
>> >> to look at and probably fix marked by XXX
>> >> For those lines I either used a trick that could break at any update,
>> >> or I had to assume things that I am not sure should be assumed.
>> >>
>> >> Look forward to seeing what you do with it :)
>> >>
>> >> Charles
>> >>
>> >> On Sat, Jul 3, 2010 at 3:02 AM, holger krekel <holger at merlinux.eu> wrote:
>> >> > Hi Charles,
>> >> >
>> >> > On Fri, Jul 02, 2010 at 16:31 -0500, Charles Solar wrote:
>> >> >> Ah, a few hours after sending this I found the scripts folder, and here I
>> >> >> thought socketserver.py was just an example.. oh well.
>> >> >
>> >> > Your code still makes sense - an installable windows service would be great.
>> >> >
>> >> >> Anyway after using this a bit I am afraid I need to modify how the
>> >> >> Socketgateway works and I would like a tip or two about a couple things.
>> >> >>
>> >> >> I noticed that socketgateways will run the remote code in their own world,
>> >> >> which is a bit different from ssh gateways where the ssh session terminates
>> >> >> when the job is done. ?The sshd remains untainted because the remote server
>> >> >> spawned a nice python process for us.
>> >> >> However the socketgateways run the remote code in themselves and thus if the
>> >> >> remote code dirties up the interpreter the whole daemon is bad.
>> >> >
>> >> > True.
>> >> >
>> >> >> In my case I am starting a twisted reactor on the remote server, and once a
>> >> >> twisted reactor has been created its expected that it is alive as long as
>> >> >> the python process itself. ?Because of this, the socket gateway daemon is
>> >> >> only good for 1 connection, then it crashes because twisted's reactor has
>> >> >> issues.
>> >> >>
>> >> >> The solution to this problem would be to execute the incoming remote code in
>> >> >> a new popen gateway on the remote server instead of inside the socket
>> >> >> gateway instance itself. ?I have been skimming a few files but I am not
>> >> >> completely sure how or where would be a good place to put the popen gateway.
>> >> >> My gut tells me the proxy should be put in SocketIO, but I figured it might
>> >> >> save some time to send out an email to find out how these systems interact.
>> >> >
>> >> > not sure, but for now i'd rather not try to nest execnet gateways,
>> >> > mostly because it will be fun to debug (there is some logging and
>> >> > nested gateways generally work though). ?Also, a nice property of
>> >> > the socketgateway server is that it's rather independent from execnet
>> >> > impl details.
>> >> >
>> >> > Rather, the socketgateway.py service could learn to act just like
>> >> > the ssh-daemon by using subprocess.Popen and allowing multiple connections.
>> >> > IMHO just using threads that own a socket and proxy IO to their subprocess'ed
>> >> > gateway could be straightforward.
>> >> >
>> >> > In any case, I am happy to review and integrate your code into the next release
>> >> > if you go down this route, both for the service and the socketserver issue.
>> >> >
>> >> > cheers,
>> >> > holger
>> >> >
>> >> >
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:
>> >> >>
>> >> >> > I wrote up a windows service script for starting a socket gateway. ?Thought
>> >> >> > other people might like to use it. ?Its pretty basic, but gets the job
>> >> >> > done. ?PyWin32 is required.
>> >> >> >
>> >> >> > Thanks for the great library.
>> >> >> >
>> >> >> > Charles
>> >> >> >
>> >> >
>> >> >> _______________________________________________
>> >> >> execnet-dev mailing list
>> >> >> execnet-dev at codespeak.net
>> >> >> http://codespeak.net/mailman/listinfo/execnet-dev
>> >> >
>> >> >
>> >> > --
>> >> >
>> >
>> >> """
>> >> Windows service for handling incomming socket gateways
>> >> """
>> >>
>> >> import threading, SocketServer
>> >>
>> >> import win32serviceutil
>> >> import win32service
>> >> import win32event
>> >> import servicemanager
>> >>
>> >> import execnet
>> >> from execnet.gateway_socket import SocketIO
>> >>
>> >> class IOJoiner():
>> >> ? ? """
>> >> ? ? Joins two io instances so when one wants to read, the incoming data is
>> >> ? ? sent straight to the other io. ?Useful for tieing the Popen gateway IO
>> >> ? ? and the SocketIO.
>> >> ? ? """
>> >> ? ? def __init__( self, primaryIO, secondaryIO ):
>> >> ? ? ? ? self.prim = primaryIO
>> >> ? ? ? ? self.sec = secondaryIO
>> >>
>> >> ? ? def read( self, numbytes ):
>> >> ? ? ? ? buf = self.prim.read( numbytes )
>> >> ? ? ? ? self.sec.write( buf )
>> >> ? ? ? ? # 'L' corresponds to the 'NoneType' in the Unserializer
>> >> ? ? ? ? # Do this so the person who called read does not freak out
>> >> ? ? ? ? # XXX I should return an object ACTUALLY representing NoneType, but not sure atm
>> >> ? ? ? ? # how to access that data.
>> >> ? ? ? ? return 'L'
>> >>
>> >> ? ? def write( self, data ):
>> >> ? ? ? ? # XXX I do not see any reason for the other io to worry
>> >> ? ? ? ? # About things the primary is writing, I could be wrong though.
>> >> ? ? ? ? self.prim.write( data )
>> >>
>> >> ? ? def close_read( self ):
>> >> ? ? ? ? self.prim.close_read()
>> >>
>> >> ? ? def close_write( self ):
>> >> ? ? ? ? self.prim.close_write()
>> >>
>> >>
>> >> class PythonService(win32serviceutil.ServiceFramework):
>> >> ? ? _svc_name_ = "PythonSocketServer"
>> >> ? ? _svc_display_name_ = "Python execnet socket server"
>> >> ? ? _svc_description_ = "Gateway server to allow execnet to connect to windows machines"
>> >> ? ? def __init__(self, args):
>> >> ? ? ? ? win32serviceutil.ServiceFramework.__init__(self, args)
>> >> ? ? ? ? # Create an event which we will use to wait on.
>> >> ? ? ? ? # The "service stop" request will set this event.
>> >> ? ? ? ? self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
>> >>
>> >> ? ? def SvcStop(self):
>> >> ? ? ? ? # Before we do anything, tell the SCM we are starting the stop process.
>> >> ? ? ? ? self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
>> >> ? ? ? ? # And set my event.
>> >> ? ? ? ? win32event.SetEvent(self.hWaitStop)
>> >>
>> >> ? ? def SvcDoRun(self):
>> >>
>> >> ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STARTED,(self._svc_name_, ''))
>> >>
>> >> ? ? ? ? self.main()
>> >>
>> >> ? ? ? ? while True:
>> >> ? ? ? ? ? ? rc = win32event.WaitForSingleObject(self.hWaitStop, 1000)
>> >> ? ? ? ? ? ? if rc == win32event.WAIT_OBJECT_0:
>> >> ? ? ? ? ? ? ? ? self._server.shutdown()
>> >> ? ? ? ? ? ? ? ? self._serverThread.join()
>> >> ? ? ? ? ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STOPPED,(self._svc_name_, ''))
>> >> ? ? ? ? ? ? ? ? break
>> >>
>> >> ? ? def main( self ):
>> >>
>> >> ? ? ? ? class TCPHandler( SocketServer.StreamRequestHandler ):
>> >> ? ? ? ? ? ? def handle( self ):
>> >> ? ? ? ? ? ? ? ? source = self.rfile.readline().rstrip()
>> >>
>> >> ? ? ? ? ? ? ? ? # Spawn a child python process for the new connection
>> >> ? ? ? ? ? ? ? ? # XXX It would be nice if there was a way to remotely request a new ironpython or jython instance instead of a generic cpython instance
>> >> ? ? ? ? ? ? ? ? # but that would require modifications to SocketGateway I believe.
>> >> ? ? ? ? ? ? ? ? gw = execnet.PopenGateway(python='python')
>> >>
>> >> ? ? ? ? ? ? ? ? sockio = SocketIO( self.request )
>> >> ? ? ? ? ? ? ? ? gwio = gw._io
>> >>
>> >> ? ? ? ? ? ? ? ? source = source.replace( "io = SocketIO(clientsock)", "io = joiner" )
>> >>
>> >> ? ? ? ? ? ? ? ? g = { 'joiner': IOJoiner( sockio, gwio ), 'address': self.client_address }
>> >> ? ? ? ? ? ? ? ? gw._io = IOJoiner( gwio, sockio )
>> >>
>> >> ? ? ? ? ? ? ? ? source = eval(source)
>> >>
>> >> ? ? ? ? ? ? ? ? if source:
>> >> ? ? ? ? ? ? ? ? ? ? co = compile( source+'\n', source, 'exec' )
>> >>
>> >> ? ? ? ? ? ? ? ? ? ? try:
>> >> ? ? ? ? ? ? ? ? ? ? ? ? exec co in g
>> >> ? ? ? ? ? ? ? ? ? ? except Exception as e:
>> >> ? ? ? ? ? ? ? ? ? ? ? ? servicemanager.LogErrorMsg( "Execution of received source code raised the following exception: %r" % e )
>> >>
>> >> ? ? ? ? self._server = SocketServer.ThreadingTCPServer( ('0.0.0.0', 8888), TCPHandler )
>> >> ? ? ? ? self._serverThread = threading.Thread( target=self._server.serve_forever )
>> >> ? ? ? ? self._serverThread.setDaemon( True )
>> >> ? ? ? ? self._serverThread.start()
>> >>
>> >> if __name__ == '__main__':
>> >> ? ? win32serviceutil.HandleCommandLine(PythonService)
>> >
>> >
>> > --
>> >
>>
>
> --
>


From holger at merlinux.eu  Mon Jul 12 09:47:53 2010
From: holger at merlinux.eu (holger krekel)
Date: Mon, 12 Jul 2010 09:47:53 +0200
Subject: [execnet-dev] Socket gateway windows service
In-Reply-To: <AANLkTim7qXuJg2FjvEq5k5C6gbY1Pg8HVl87YzxTgyV2@mail.gmail.com>
References: <AANLkTik2XRf1bDO3_hdfu5BZqhBcy5ZiSC_okUgkMA99@mail.gmail.com>
	<AANLkTilHoFMYDDINd-AVXRDSwa2_5qsF1XQcepAmqh11@mail.gmail.com>
	<20100703080159.GD14601@trillke.net>
	<AANLkTinsiOYlwwSnSajEzmdbebIt66I3RFpuvQZ6xtex@mail.gmail.com>
	<20100708163358.GR14601@trillke.net>
	<AANLkTimV-bJJq_-53oR2KUt92eDNmcmYVAdZRvhylqEC@mail.gmail.com>
	<20100711120102.GV14601@trillke.net>
	<AANLkTim7qXuJg2FjvEq5k5C6gbY1Pg8HVl87YzxTgyV2@mail.gmail.com>
Message-ID: <20100712074753.GZ14601@trillke.net>

Hi Charles, 

very good.  Let's see about the rsync-performance issue after
we tackled doing the new-style mediators.  They involve one less 
gateway and thus one less receiver-thread and thus much less indirection 
between Invocation and Worker, i think.  Subprocess-pipes are quite fast usually
so i wouldn't expect much of a slow down.  Btw, I am myself not able to do much
coding before some time after EuroPython 2010 - are you subscribed to 
execnet-commit?  

If you want to go ahead with some internal refactorings just clone 
the execnet repository and send me (pull) requests for review. 

cheers,
holger

On Sun, Jul 11, 2010 at 12:01 -0500, Charles Solar wrote:
> Comments inline for this one :)
> 
> On Sun, Jul 11, 2010 at 7:01 AM, holger krekel <holger at merlinux.eu> wrote:
> > Hi Charles,
> >
> > On Thu, Jul 08, 2010 at 12:03 -0500, Charles Solar wrote:
> >> Well its a windows service so on any windows machine you would just do
> >>
> >> python socketserver.py install
> >> python socketserver.py start
> >>
> >> to get the socketserver running on port 8888.
> >
> > i've never seen or done this, actually. ?Thanks :)
> >
> >> You can connect to it from any machine with
> >> import execnet
> >> gw=execnet.makegateway( "socket=[hostname]:8888" )
> >>
> >> on the socket server it will spawn a new popen gateway and connect the
> >> socketIO to the popenIO through IOJoiner.
> >>
> >> IOJoiner replaces the io that the gateways on the server side use to
> >> serve requests. ?So when the slave socket gateway calls io.read(1) in
> >> the serve loop, its actually calling IOJoiner.read(). ?IOJoiner.read()
> >> will then read from the socket, and write all incoming data to the
> >> popen gateway.
> >
> > Let me introduce some terms for the three processes in question:
> >
> > ? ?Invocation ? ? ? ? Mediator ? ? ?Worker
> > ? ? ? ? ? ? ?<socket> ? ? ? ? ?<popen>
> >
> > And then try to rephrase what you said to see if i understood:
> > The Mediator and Invocation process use a SocketGateway
> > and the Worker and Mediator a PopenGateway. ?The Mediator
> > replaces both the Mediator-Invocation IO and and the Mediator-Worker
> > IO with Joiner instances.
> >
> > Then you run the SocketGateway serve code which reads from the
> > Mediator-Invocation Joiner-IO and forward the raw data
> > to the *original* Worker-IO. ?The Receiver-Thread of the Mediator-Worker
> > PopenGateway also uses the joiner and what it reads it sends
> > to the original Invocation IO.
> 
> yep, absolutely correct.
> 
> >
> > Requires some concentration to think about this :)
> 
> took me several hours of meditation to come up with it haha.
> 
> >
> > This method you are using requires that the socketservice side
> > has execnet installed and is a bit fragile wrt to execnet
> > integration (the "L" bit although we could maybe introduce a
> > special exception for that). Cool that it works :)
> >
> > I am not sure yet how an alternative could look like.
> > Here are my current thoughts helping also myself
> > to get clearer.
> >
> > I'd like to have something higher level, maybe introduce
> > internally the concept of an "Instantiator", that would
> > get called to "open a subprocess with a bootstrap" and
> > return an IO object. ?It would be naturally used by
> > popen-style gateways. This internal refactoring is
> > just an increment how things are now.
> >
> > We could then extend the instantiator to make it
> > call "subprocess.Popen" not directly but rather
> > through an existing Gateway, returning an IO
> > object on the invocation side that is based
> > on communication with the mediator through the/a channel.
> >
> > For bootstrapping a new gateway the Invocation/instantiator
> > would send an "open a subprocess with bootstrap" command
> > to a loop which it installed on the Mediator.
> > The Mediator would call "subprocess.Popen" with the
> > given bootstrap and then read/write from the subprocess
> > pipes and read/write to the channel which connects
> > the Invocation and Mediator.
> 
> Yea that is where I was stuck because I could not modify what the
> invoker to send a different bootstrap to the socket server.  I
> actually had to do a string replace with the given bootstrap code to
> make it use the joiner class.  Which I really did not like doing.
> 
> >
> > If i see it correctly the Mediator would basically
> > do popen.stdin.read(n) and if it reads something
> > write it to the Invocation/Mediator channel.
> > And if the Mediator receives data a channel callback
> > (usually running in the receiver-thread)
> > could send it to Popen.write(s).
> >
> > This way execnet would not need to be installed
> > at the Mediator side and the concept should work
> > uniformly for all gateway types ...
> >
> > cheers,
> > holger
> >
> 
> Sounds good but I have to go digging around in the code more to
> understand where things are connected more clearly.  I was limited by
> not wanting to change execnet internally but if you think this can be
> done with a minimal change to add new features to the popen gateway
> then that sounds like the best route.
> There is something I found out while using my solution the other day.
> I was using RSync to a machine using this socket service and the file
> transfers were taking minutes to send files of 1-2 megs in size.  It
> appears this io mediation takes its toll on communication speed.
> Because for rsync its not necessary to spawn a child process and
> communicate through pipes, I decided to open a new socket that acts
> like the original socket server and executes the commands without
> spawning a child process.  Which fixes the rsync issue.
> I mention it because while we are refactoring this io a bit it might
> be a good idea to make a signal for 'dont spawn child process' or
> something.
> 
> Charles
> 
> >> However, the socket slave gateway still wants a return value.
> >>
> >> Returning what we just read from the socket will produce incorrect
> >> results, so I have it just return the Unserializer's None type, which
> >> will cause the serve loop to not do anything, exactly what I wanted.
> >>
> >> On the popen end, I write the incoming socket data to the master
> >> gateway's io, which is then read by the slave gateway and used. ?Also,
> >> all data read from the popen slave gateway is written to the socket
> >> io, same as how socket io passes to popen io.
> >>
> >> IOJoiner needed to use Unserializer types because I did not want the
> >> slave socket gateway or the master popen io do anything, as their only
> >> purpose is to facilitate communication between the other io gateways.
> >>
> >> I have no doubt there is a better way to do this, but with my limited
> >> understanding of execnet internals at the moment, this seems like the
> >> easiest solution.
> >>
> >> Charles
> >>
> >> On Thu, Jul 8, 2010 at 11:33 AM, holger krekel <holger at merlinux.eu> wrote:
> >> > Hi Charles,
> >> >
> >> > am quite busy with releasing some bits before i leave for a couple of days
> >> > but i am wondering:
> >> >
> >> > a) could you provide a full example on how to use your code?
> >> >
> >> > b) why does the IOJoiner need to be aware of (Un)Serializer details?
> >> >
> >> > cheers,
> >> > holger
> >> >
> >> > On Thu, Jul 08, 2010 at 10:39 -0500, Charles Solar wrote:
> >> >> Ok I have a solution that works for my purposes. ?I rethought my
> >> >> original idea and figured out how to proxy the io instances to the new
> >> >> popengateway that socketserver creates, so no modifications to execnet
> >> >> internals was necessary.
> >> >> I am attaching my solution, but there are some lines I would like you
> >> >> to look at and probably fix marked by XXX
> >> >> For those lines I either used a trick that could break at any update,
> >> >> or I had to assume things that I am not sure should be assumed.
> >> >>
> >> >> Look forward to seeing what you do with it :)
> >> >>
> >> >> Charles
> >> >>
> >> >> On Sat, Jul 3, 2010 at 3:02 AM, holger krekel <holger at merlinux.eu> wrote:
> >> >> > Hi Charles,
> >> >> >
> >> >> > On Fri, Jul 02, 2010 at 16:31 -0500, Charles Solar wrote:
> >> >> >> Ah, a few hours after sending this I found the scripts folder, and here I
> >> >> >> thought socketserver.py was just an example.. oh well.
> >> >> >
> >> >> > Your code still makes sense - an installable windows service would be great.
> >> >> >
> >> >> >> Anyway after using this a bit I am afraid I need to modify how the
> >> >> >> Socketgateway works and I would like a tip or two about a couple things.
> >> >> >>
> >> >> >> I noticed that socketgateways will run the remote code in their own world,
> >> >> >> which is a bit different from ssh gateways where the ssh session terminates
> >> >> >> when the job is done. ?The sshd remains untainted because the remote server
> >> >> >> spawned a nice python process for us.
> >> >> >> However the socketgateways run the remote code in themselves and thus if the
> >> >> >> remote code dirties up the interpreter the whole daemon is bad.
> >> >> >
> >> >> > True.
> >> >> >
> >> >> >> In my case I am starting a twisted reactor on the remote server, and once a
> >> >> >> twisted reactor has been created its expected that it is alive as long as
> >> >> >> the python process itself. ?Because of this, the socket gateway daemon is
> >> >> >> only good for 1 connection, then it crashes because twisted's reactor has
> >> >> >> issues.
> >> >> >>
> >> >> >> The solution to this problem would be to execute the incoming remote code in
> >> >> >> a new popen gateway on the remote server instead of inside the socket
> >> >> >> gateway instance itself. ?I have been skimming a few files but I am not
> >> >> >> completely sure how or where would be a good place to put the popen gateway.
> >> >> >> My gut tells me the proxy should be put in SocketIO, but I figured it might
> >> >> >> save some time to send out an email to find out how these systems interact.
> >> >> >
> >> >> > not sure, but for now i'd rather not try to nest execnet gateways,
> >> >> > mostly because it will be fun to debug (there is some logging and
> >> >> > nested gateways generally work though). ?Also, a nice property of
> >> >> > the socketgateway server is that it's rather independent from execnet
> >> >> > impl details.
> >> >> >
> >> >> > Rather, the socketgateway.py service could learn to act just like
> >> >> > the ssh-daemon by using subprocess.Popen and allowing multiple connections.
> >> >> > IMHO just using threads that own a socket and proxy IO to their subprocess'ed
> >> >> > gateway could be straightforward.
> >> >> >
> >> >> > In any case, I am happy to review and integrate your code into the next release
> >> >> > if you go down this route, both for the service and the socketserver issue.
> >> >> >
> >> >> > cheers,
> >> >> > holger
> >> >> >
> >> >> >
> >> >> >>
> >> >> >> Thanks
> >> >> >>
> >> >> >> On Fri, Jul 2, 2010 at 11:31 AM, Charles Solar <charlessolar at gmail.com>wrote:
> >> >> >>
> >> >> >> > I wrote up a windows service script for starting a socket gateway. ?Thought
> >> >> >> > other people might like to use it. ?Its pretty basic, but gets the job
> >> >> >> > done. ?PyWin32 is required.
> >> >> >> >
> >> >> >> > Thanks for the great library.
> >> >> >> >
> >> >> >> > Charles
> >> >> >> >
> >> >> >
> >> >> >> _______________________________________________
> >> >> >> execnet-dev mailing list
> >> >> >> execnet-dev at codespeak.net
> >> >> >> http://codespeak.net/mailman/listinfo/execnet-dev
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >
> >> >> """
> >> >> Windows service for handling incomming socket gateways
> >> >> """
> >> >>
> >> >> import threading, SocketServer
> >> >>
> >> >> import win32serviceutil
> >> >> import win32service
> >> >> import win32event
> >> >> import servicemanager
> >> >>
> >> >> import execnet
> >> >> from execnet.gateway_socket import SocketIO
> >> >>
> >> >> class IOJoiner():
> >> >> ? ? """
> >> >> ? ? Joins two io instances so when one wants to read, the incoming data is
> >> >> ? ? sent straight to the other io. ?Useful for tieing the Popen gateway IO
> >> >> ? ? and the SocketIO.
> >> >> ? ? """
> >> >> ? ? def __init__( self, primaryIO, secondaryIO ):
> >> >> ? ? ? ? self.prim = primaryIO
> >> >> ? ? ? ? self.sec = secondaryIO
> >> >>
> >> >> ? ? def read( self, numbytes ):
> >> >> ? ? ? ? buf = self.prim.read( numbytes )
> >> >> ? ? ? ? self.sec.write( buf )
> >> >> ? ? ? ? # 'L' corresponds to the 'NoneType' in the Unserializer
> >> >> ? ? ? ? # Do this so the person who called read does not freak out
> >> >> ? ? ? ? # XXX I should return an object ACTUALLY representing NoneType, but not sure atm
> >> >> ? ? ? ? # how to access that data.
> >> >> ? ? ? ? return 'L'
> >> >>
> >> >> ? ? def write( self, data ):
> >> >> ? ? ? ? # XXX I do not see any reason for the other io to worry
> >> >> ? ? ? ? # About things the primary is writing, I could be wrong though.
> >> >> ? ? ? ? self.prim.write( data )
> >> >>
> >> >> ? ? def close_read( self ):
> >> >> ? ? ? ? self.prim.close_read()
> >> >>
> >> >> ? ? def close_write( self ):
> >> >> ? ? ? ? self.prim.close_write()
> >> >>
> >> >>
> >> >> class PythonService(win32serviceutil.ServiceFramework):
> >> >> ? ? _svc_name_ = "PythonSocketServer"
> >> >> ? ? _svc_display_name_ = "Python execnet socket server"
> >> >> ? ? _svc_description_ = "Gateway server to allow execnet to connect to windows machines"
> >> >> ? ? def __init__(self, args):
> >> >> ? ? ? ? win32serviceutil.ServiceFramework.__init__(self, args)
> >> >> ? ? ? ? # Create an event which we will use to wait on.
> >> >> ? ? ? ? # The "service stop" request will set this event.
> >> >> ? ? ? ? self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
> >> >>
> >> >> ? ? def SvcStop(self):
> >> >> ? ? ? ? # Before we do anything, tell the SCM we are starting the stop process.
> >> >> ? ? ? ? self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
> >> >> ? ? ? ? # And set my event.
> >> >> ? ? ? ? win32event.SetEvent(self.hWaitStop)
> >> >>
> >> >> ? ? def SvcDoRun(self):
> >> >>
> >> >> ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STARTED,(self._svc_name_, ''))
> >> >>
> >> >> ? ? ? ? self.main()
> >> >>
> >> >> ? ? ? ? while True:
> >> >> ? ? ? ? ? ? rc = win32event.WaitForSingleObject(self.hWaitStop, 1000)
> >> >> ? ? ? ? ? ? if rc == win32event.WAIT_OBJECT_0:
> >> >> ? ? ? ? ? ? ? ? self._server.shutdown()
> >> >> ? ? ? ? ? ? ? ? self._serverThread.join()
> >> >> ? ? ? ? ? ? ? ? servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,servicemanager.PYS_SERVICE_STOPPED,(self._svc_name_, ''))
> >> >> ? ? ? ? ? ? ? ? break
> >> >>
> >> >> ? ? def main( self ):
> >> >>
> >> >> ? ? ? ? class TCPHandler( SocketServer.StreamRequestHandler ):
> >> >> ? ? ? ? ? ? def handle( self ):
> >> >> ? ? ? ? ? ? ? ? source = self.rfile.readline().rstrip()
> >> >>
> >> >> ? ? ? ? ? ? ? ? # Spawn a child python process for the new connection
> >> >> ? ? ? ? ? ? ? ? # XXX It would be nice if there was a way to remotely request a new ironpython or jython instance instead of a generic cpython instance
> >> >> ? ? ? ? ? ? ? ? # but that would require modifications to SocketGateway I believe.
> >> >> ? ? ? ? ? ? ? ? gw = execnet.PopenGateway(python='python')
> >> >>
> >> >> ? ? ? ? ? ? ? ? sockio = SocketIO( self.request )
> >> >> ? ? ? ? ? ? ? ? gwio = gw._io
> >> >>
> >> >> ? ? ? ? ? ? ? ? source = source.replace( "io = SocketIO(clientsock)", "io = joiner" )
> >> >>
> >> >> ? ? ? ? ? ? ? ? g = { 'joiner': IOJoiner( sockio, gwio ), 'address': self.client_address }
> >> >> ? ? ? ? ? ? ? ? gw._io = IOJoiner( gwio, sockio )
> >> >>
> >> >> ? ? ? ? ? ? ? ? source = eval(source)
> >> >>
> >> >> ? ? ? ? ? ? ? ? if source:
> >> >> ? ? ? ? ? ? ? ? ? ? co = compile( source+'\n', source, 'exec' )
> >> >>
> >> >> ? ? ? ? ? ? ? ? ? ? try:
> >> >> ? ? ? ? ? ? ? ? ? ? ? ? exec co in g
> >> >> ? ? ? ? ? ? ? ? ? ? except Exception as e:
> >> >> ? ? ? ? ? ? ? ? ? ? ? ? servicemanager.LogErrorMsg( "Execution of received source code raised the following exception: %r" % e )
> >> >>
> >> >> ? ? ? ? self._server = SocketServer.ThreadingTCPServer( ('0.0.0.0', 8888), TCPHandler )
> >> >> ? ? ? ? self._serverThread = threading.Thread( target=self._server.serve_forever )
> >> >> ? ? ? ? self._serverThread.setDaemon( True )
> >> >> ? ? ? ? self._serverThread.start()
> >> >>
> >> >> if __name__ == '__main__':
> >> >> ? ? win32serviceutil.HandleCommandLine(PythonService)
> >> >
> >> >
> >> > --
> >> >
> >>
> >
> > --
> >
> 

-- 


From charlessolar at gmail.com  Tue Jul 13 00:16:30 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Mon, 12 Jul 2010 17:16:30 -0500
Subject: [execnet-dev] RSync Issue
In-Reply-To: <20100711103618.GT14601@trillke.net>
References: <AANLkTilS-Xxx1uxslsFvb0Wpu9bOJZjm-dFtGpPHkFJm@mail.gmail.com> 
	<20100711103618.GT14601@trillke.net>
Message-ID: <AANLkTikXHQd1PUR8DGSnpveOqkHKWkXpi1SzNOGRGEIF@mail.gmail.com>

I'll make up a small test case but just so I do not forgot I found
another bug dealing with symlinks.
I think that all relative symlinks are broken, or else I do not
understand how they could have worked before.
I saw it be having a structure like this rsync:

lib/
   libnet.so -> libnet.so.1.0.0
   libnet.so.1.0.0

What it created on the remote side was

lib/
  libnet.so -> /tmp/rsynctemp/libnet.so.1.0.0
  libnet.so.1.0.0

The link on the remote side did not include the lib subdir in the
link.  After looking at the code I fixed it up but I do not think they
should be working at all since the code atm only works if the symlink
is in the root dest directory.

Ill send in a patch for you to look at.

Charles

On Sun, Jul 11, 2010 at 5:36 AM, holger krekel <holger at merlinux.eu> wrote:
> Hi Charles,
>
> On Fri, Jul 09, 2010 at 10:03 -0500, Charles Solar wrote:
>> It would appear that RSync will only work with brand new gateways that
>> have not had any other operations run on them. ?Is this intentional?
>
> No. ?Sounds like a bug.
>
>> For example, I do something like
>>
>> gw = execnet.makegateway( "ssh=foobar" )
>> gw.remote_exec( mybootstrapper )
>>
>> ... then later I want to rsync ...
>>
>> rsync = execnet.RSync( "/tmp/foobar" )
>> rsync.add_target( gw, "/tmp/barfoo" )
>> rsync.send()
>>
>> I get remote EOFErrors
>>
>> File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
>> ? ? self._end_of_channel(channel)
>> ? File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
>> ? ? channel.waitclose()
>> ? File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
>> ? ? raise error
>> exceptions.EOFError: expected 1 bytes, got 0
>>
>> If I however create a brand new gateway to give to rsync it works fine.
>>
>> I would think that since rsync just creates a new channel on the
>> gateway that it should work on existing 'old' gateways.
>
> yip. Myself i only used it for doing things at the beginning, it seems.
> Do you have or could you try to find an repeatable minimal example?
>
> best,
>
> holger
>


From holger at merlinux.eu  Tue Jul 13 08:49:03 2010
From: holger at merlinux.eu (holger krekel)
Date: Tue, 13 Jul 2010 08:49:03 +0200
Subject: [execnet-dev] RSync Issue
In-Reply-To: <AANLkTikXHQd1PUR8DGSnpveOqkHKWkXpi1SzNOGRGEIF@mail.gmail.com>
References: <AANLkTilS-Xxx1uxslsFvb0Wpu9bOJZjm-dFtGpPHkFJm@mail.gmail.com>
	<20100711103618.GT14601@trillke.net>
	<AANLkTikXHQd1PUR8DGSnpveOqkHKWkXpi1SzNOGRGEIF@mail.gmail.com>
Message-ID: <20100713064903.GG14601@trillke.net>

Hi Charles, 

it's a known issue but was reported for the "pytest-xdist" plugin:

    http://bitbucket.org/hpk42/py-trunk/issue/70/distributed-testing-rsync-mangles-some-sym

I think RSync needs some work, there also is this:

    http://bitbucket.org/hpk42/py-trunk/issue/68/distributed-testing-rsync-does-not-maintain-file

In general i am not sure it makes sense to maintain the added complexity 
that arises from the fact that execnet.RSync tries to do 1:N rsyncing
efficiently (i.e. syncing from host to multiple others at the same time). 
This only works well anyway if the destination hosts have similar network
characteristics.  Maybe it makes sense to just copy and simplify the
code and release a 1:1 more advanced rsync utility depending on the 
execnet package and deprecating the old one. 

I haven't looked at the issue, maybe it can also just simply get fixed :)

best,
holger


On Mon, Jul 12, 2010 at 17:16 -0500, Charles Solar wrote:
> I'll make up a small test case but just so I do not forgot I found
> another bug dealing with symlinks.
> I think that all relative symlinks are broken, or else I do not
> understand how they could have worked before.
> I saw it be having a structure like this rsync:
> 
> lib/
>    libnet.so -> libnet.so.1.0.0
>    libnet.so.1.0.0
> 
> What it created on the remote side was
> 
> lib/
>   libnet.so -> /tmp/rsynctemp/libnet.so.1.0.0
>   libnet.so.1.0.0
> 
> The link on the remote side did not include the lib subdir in the
> link.  After looking at the code I fixed it up but I do not think they
> should be working at all since the code atm only works if the symlink
> is in the root dest directory.
> 
> Ill send in a patch for you to look at.
> 
> Charles
> 
> On Sun, Jul 11, 2010 at 5:36 AM, holger krekel <holger at merlinux.eu> wrote:
> > Hi Charles,
> >
> > On Fri, Jul 09, 2010 at 10:03 -0500, Charles Solar wrote:
> >> It would appear that RSync will only work with brand new gateways that
> >> have not had any other operations run on them. ?Is this intentional?
> >
> > No. ?Sounds like a bug.
> >
> >> For example, I do something like
> >>
> >> gw = execnet.makegateway( "ssh=foobar" )
> >> gw.remote_exec( mybootstrapper )
> >>
> >> ... then later I want to rsync ...
> >>
> >> rsync = execnet.RSync( "/tmp/foobar" )
> >> rsync.add_target( gw, "/tmp/barfoo" )
> >> rsync.send()
> >>
> >> I get remote EOFErrors
> >>
> >> File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
> >> ? ? self._end_of_channel(channel)
> >> ? File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
> >> ? ? channel.waitclose()
> >> ? File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
> >> ? ? raise error
> >> exceptions.EOFError: expected 1 bytes, got 0
> >>
> >> If I however create a brand new gateway to give to rsync it works fine.
> >>
> >> I would think that since rsync just creates a new channel on the
> >> gateway that it should work on existing 'old' gateways.
> >
> > yip. Myself i only used it for doing things at the beginning, it seems.
> > Do you have or could you try to find an repeatable minimal example?
> >
> > best,
> >
> > holger
> >
> 

-- 


From newen88 at gmail.com  Wed Jul 14 18:31:28 2010
From: newen88 at gmail.com (Eduardo Coelho)
Date: Wed, 14 Jul 2010 13:31:28 -0300
Subject: [execnet-dev] Eclipse using multiple Python interpreters with
	execnet
Message-ID: <AANLkTilKuo4_xhk1rzLeVMFCOCBqB3LVwIiIUr7iYMe0@mail.gmail.com>

Hi folks, I asked a question about this topic @ stackoverflow.
But I realized asking this here would be good too.

I can use the execnet package normally through the bash Terminal, but
inside my Eclipse project I've got some errors while trying to mix
different Python Interpreters. It sounds more like a Eclipse
configuration problem than the execnet package itself, but I hope
people in this mailing-list are capable to give me some instructions
on how to deal with it.

In order to no post redundant information, I'll follow the
stackoverflow question's link:

http://stackoverflow.com/questions/3248271/eclipse-using-multiple-python-interpreters-with-execnet

Appreciate any replies.
Thanks

-- 
Live as if you were to die tomorrow. Learn as if you were to live forever.

Eduardo Coelho


From execnet-dev at codespeak.net  Thu Jul 15 06:16:44 2010
From: execnet-dev at codespeak.net (execnet-dev at codespeak.net)
Date: Thu, 15 Jul 2010 06:16:44 +0200 (CEST)
Subject: execnet-dev@codespeak.net VIAGRA ® Official Site 76% OFF
Message-ID: <20100715041644.8FDB4282BFC@codespeak.net>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100715/30ae6239/attachment.html>

From holger at merlinux.eu  Thu Jul 15 10:40:05 2010
From: holger at merlinux.eu (holger krekel)
Date: Thu, 15 Jul 2010 10:40:05 +0200
Subject: [execnet-dev] Eclipse using multiple Python interpreters
	with	execnet
In-Reply-To: <AANLkTilKuo4_xhk1rzLeVMFCOCBqB3LVwIiIUr7iYMe0@mail.gmail.com>
References: <AANLkTilKuo4_xhk1rzLeVMFCOCBqB3LVwIiIUr7iYMe0@mail.gmail.com>
Message-ID: <20100715084005.GU14601@trillke.net>

Hi Eduardo,

i gave my advise there.  your env-setting info was helpful.

best,
holger

On Wed, Jul 14, 2010 at 13:31 -0300, Eduardo Coelho wrote:
> Hi folks, I asked a question about this topic @ stackoverflow.
> But I realized asking this here would be good too.
> 
> I can use the execnet package normally through the bash Terminal, but
> inside my Eclipse project I've got some errors while trying to mix
> different Python Interpreters. It sounds more like a Eclipse
> configuration problem than the execnet package itself, but I hope
> people in this mailing-list are capable to give me some instructions
> on how to deal with it.
> 
> In order to no post redundant information, I'll follow the
> stackoverflow question's link:
> 
> http://stackoverflow.com/questions/3248271/eclipse-using-multiple-python-interpreters-with-execnet
> 
> Appreciate any replies.
> Thanks
> 
> -- 
> Live as if you were to die tomorrow. Learn as if you were to live forever.
> 
> Eduardo Coelho
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev
> 

-- 


From newen88 at gmail.com  Thu Jul 15 18:23:22 2010
From: newen88 at gmail.com (Eduardo Coelho)
Date: Thu, 15 Jul 2010 13:23:22 -0300
Subject: [execnet-dev] Eclipse using multiple Python interpreters with
	execnet
In-Reply-To: <20100715084005.GU14601@trillke.net>
References: <AANLkTilKuo4_xhk1rzLeVMFCOCBqB3LVwIiIUr7iYMe0@mail.gmail.com>
	<20100715084005.GU14601@trillke.net>
Message-ID: <AANLkTikKRt_WZCKu6Lp8GNdfpejDK4zxRrzl5G6J4Cpr@mail.gmail.com>

Thanks holger!!!

On Thu, Jul 15, 2010 at 5:40 AM, holger krekel <holger at merlinux.eu> wrote:
> Hi Eduardo,
>
> i gave my advise there. ?your env-setting info was helpful.
>
> best,
> holger
>
> On Wed, Jul 14, 2010 at 13:31 -0300, Eduardo Coelho wrote:
>> Hi folks, I asked a question about this topic @ stackoverflow.
>> But I realized asking this here would be good too.
>>
>> I can use the execnet package normally through the bash Terminal, but
>> inside my Eclipse project I've got some errors while trying to mix
>> different Python Interpreters. It sounds more like a Eclipse
>> configuration problem than the execnet package itself, but I hope
>> people in this mailing-list are capable to give me some instructions
>> on how to deal with it.
>>
>> In order to no post redundant information, I'll follow the
>> stackoverflow question's link:
>>
>> http://stackoverflow.com/questions/3248271/eclipse-using-multiple-python-interpreters-with-execnet
>>
>> Appreciate any replies.
>> Thanks
>>
>> --
>> Live as if you were to die tomorrow. Learn as if you were to live forever.
>>
>> Eduardo Coelho
>> _______________________________________________
>> execnet-dev mailing list
>> execnet-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/execnet-dev
>>
>
> --
>



-- 
Live as if you were to die tomorrow. Learn as if you were to live forever.

Eduardo Coelho


From holger at merlinux.eu  Tue Jul 27 19:47:48 2010
From: holger at merlinux.eu (holger krekel)
Date: Tue, 27 Jul 2010 19:47:48 +0200
Subject: [execnet-dev] RSync Issue
In-Reply-To: <AANLkTikXHQd1PUR8DGSnpveOqkHKWkXpi1SzNOGRGEIF@mail.gmail.com>
References: <AANLkTilS-Xxx1uxslsFvb0Wpu9bOJZjm-dFtGpPHkFJm@mail.gmail.com>
	<20100711103618.GT14601@trillke.net>
	<AANLkTikXHQd1PUR8DGSnpveOqkHKWkXpi1SzNOGRGEIF@mail.gmail.com>
Message-ID: <20100727174748.GM14601@trillke.net>

Hi Charles, 

so i think that rsyncing symlinks should now work reaonsably well.  
I used your patches as a starting point but also added tests and made
sure permissions are also transferred for directories.  Could you verify
that after installing 

    http://hudson.testrun.org/view/pytest/job/execnet-sdist/lastSuccessfulBuild/artifact/.tox/dist/execnet-1.0.8a1.zip 

working with symlinks works for you? 

Also, i am still interested in a small failing test case for the other issue
where you could not use RSync later on. 

thanks & best,
holger

On Mon, Jul 12, 2010 at 17:16 -0500, Charles Solar wrote:
> I'll make up a small test case but just so I do not forgot I found
> another bug dealing with symlinks.
> I think that all relative symlinks are broken, or else I do not
> understand how they could have worked before.
> I saw it be having a structure like this rsync:
> 
> lib/
>    libnet.so -> libnet.so.1.0.0
>    libnet.so.1.0.0
> 
> What it created on the remote side was
> 
> lib/
>   libnet.so -> /tmp/rsynctemp/libnet.so.1.0.0
>   libnet.so.1.0.0
> 
> The link on the remote side did not include the lib subdir in the
> link.  After looking at the code I fixed it up but I do not think they
> should be working at all since the code atm only works if the symlink
> is in the root dest directory.
> 
> Ill send in a patch for you to look at.
> 
> Charles
> 
> On Sun, Jul 11, 2010 at 5:36 AM, holger krekel <holger at merlinux.eu> wrote:
> > Hi Charles,
> >
> > On Fri, Jul 09, 2010 at 10:03 -0500, Charles Solar wrote:
> >> It would appear that RSync will only work with brand new gateways that
> >> have not had any other operations run on them. ?Is this intentional?
> >
> > No. ?Sounds like a bug.
> >
> >> For example, I do something like
> >>
> >> gw = execnet.makegateway( "ssh=foobar" )
> >> gw.remote_exec( mybootstrapper )
> >>
> >> ... then later I want to rsync ...
> >>
> >> rsync = execnet.RSync( "/tmp/foobar" )
> >> rsync.add_target( gw, "/tmp/barfoo" )
> >> rsync.send()
> >>
> >> I get remote EOFErrors
> >>
> >> File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
> >> ? ? self._end_of_channel(channel)
> >> ? File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
> >> ? ? channel.waitclose()
> >> ? File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
> >> ? ? raise error
> >> exceptions.EOFError: expected 1 bytes, got 0
> >>
> >> If I however create a brand new gateway to give to rsync it works fine.
> >>
> >> I would think that since rsync just creates a new channel on the
> >> gateway that it should work on existing 'old' gateways.
> >
> > yip. Myself i only used it for doing things at the beginning, it seems.
> > Do you have or could you try to find an repeatable minimal example?
> >
> > best,
> >
> > holger
> >
> 

-- 


From matdrapeau at gmail.com  Tue Jul 27 23:36:10 2010
From: matdrapeau at gmail.com (Mathieu Drapeau)
Date: Tue, 27 Jul 2010 17:36:10 -0400
Subject: [execnet-dev] how to remotely open a program on win box?
Message-ID: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>

Hi,
I am trying to connect to a windows box and execute a program which requires
the display using execnet. The windows box has cygwin installed where sshd
is running.
Here is a small example of what I am trying to achieve.


# remote.py script which should open the windows calculator utility
from subprocess import call
if __name__ == '__channelexec__':
    call(["calc"])


#code to run locally
import execnet, remote
gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
ch = gw.remote_exec (remote)
ch.waitclose ()


Looking on the screen of my windows box, I don't see any calculator program!
Is it expected?

Thanks,
Mathieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100727/d5a869c2/attachment.html>

From charlessolar at gmail.com  Wed Jul 28 00:14:04 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Tue, 27 Jul 2010 17:14:04 -0500
Subject: [execnet-dev] RSync Issue
In-Reply-To: <20100727174748.GM14601@trillke.net>
References: <AANLkTilS-Xxx1uxslsFvb0Wpu9bOJZjm-dFtGpPHkFJm@mail.gmail.com> 
	<20100711103618.GT14601@trillke.net>
	<AANLkTikXHQd1PUR8DGSnpveOqkHKWkXpi1SzNOGRGEIF@mail.gmail.com> 
	<20100727174748.GM14601@trillke.net>
Message-ID: <AANLkTin=JyPRKW8RcDSOsng8iRtSAXKVaJiJbwhXx4d6@mail.gmail.com>

Sorry could not get around to testing this out today, i will get this
installed next time I have a chance to run my app.
Oh yea, completely forgot about the test case, I will get a small
package setup to illustrate too.

Charles

On Tue, Jul 27, 2010 at 12:47 PM, holger krekel <holger at merlinux.eu> wrote:
> Hi Charles,
>
> so i think that rsyncing symlinks should now work reaonsably well.
> I used your patches as a starting point but also added tests and made
> sure permissions are also transferred for directories. ?Could you verify
> that after installing
>
> ? ?http://hudson.testrun.org/view/pytest/job/execnet-sdist/lastSuccessfulBuild/artifact/.tox/dist/execnet-1.0.8a1.zip
>
> working with symlinks works for you?
>
> Also, i am still interested in a small failing test case for the other issue
> where you could not use RSync later on.
>
> thanks & best,
> holger
>
> On Mon, Jul 12, 2010 at 17:16 -0500, Charles Solar wrote:
>> I'll make up a small test case but just so I do not forgot I found
>> another bug dealing with symlinks.
>> I think that all relative symlinks are broken, or else I do not
>> understand how they could have worked before.
>> I saw it be having a structure like this rsync:
>>
>> lib/
>> ? ?libnet.so -> libnet.so.1.0.0
>> ? ?libnet.so.1.0.0
>>
>> What it created on the remote side was
>>
>> lib/
>> ? libnet.so -> /tmp/rsynctemp/libnet.so.1.0.0
>> ? libnet.so.1.0.0
>>
>> The link on the remote side did not include the lib subdir in the
>> link. ?After looking at the code I fixed it up but I do not think they
>> should be working at all since the code atm only works if the symlink
>> is in the root dest directory.
>>
>> Ill send in a patch for you to look at.
>>
>> Charles
>>
>> On Sun, Jul 11, 2010 at 5:36 AM, holger krekel <holger at merlinux.eu> wrote:
>> > Hi Charles,
>> >
>> > On Fri, Jul 09, 2010 at 10:03 -0500, Charles Solar wrote:
>> >> It would appear that RSync will only work with brand new gateways that
>> >> have not had any other operations run on them. ?Is this intentional?
>> >
>> > No. ?Sounds like a bug.
>> >
>> >> For example, I do something like
>> >>
>> >> gw = execnet.makegateway( "ssh=foobar" )
>> >> gw.remote_exec( mybootstrapper )
>> >>
>> >> ... then later I want to rsync ...
>> >>
>> >> rsync = execnet.RSync( "/tmp/foobar" )
>> >> rsync.add_target( gw, "/tmp/barfoo" )
>> >> rsync.send()
>> >>
>> >> I get remote EOFErrors
>> >>
>> >> File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 126, in send
>> >> ? ? self._end_of_channel(channel)
>> >> ? File "execnet-1.0.6-py2.7.egg/execnet/rsync.py", line 44, in _end_of_channel
>> >> ? ? channel.waitclose()
>> >> ? File "execnet-1.0.6-py2.7.egg/execnet/gateway_base.py", line 377, in waitclose
>> >> ? ? raise error
>> >> exceptions.EOFError: expected 1 bytes, got 0
>> >>
>> >> If I however create a brand new gateway to give to rsync it works fine.
>> >>
>> >> I would think that since rsync just creates a new channel on the
>> >> gateway that it should work on existing 'old' gateways.
>> >
>> > yip. Myself i only used it for doing things at the beginning, it seems.
>> > Do you have or could you try to find an repeatable minimal example?
>> >
>> > best,
>> >
>> > holger
>> >
>>
>
> --
>


From charlessolar at gmail.com  Wed Jul 28 00:08:30 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Tue, 27 Jul 2010 17:08:30 -0500
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>
Message-ID: <AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com>

I suspect that cygwin's sshd probably operates in non-interactive
mode.  Check the process list on the computer, does calc.exe show up?
If so, it just means that sshd is indeed non-interactive which means
it will not spawn graphical windows for logged in users.  There is
probably some way for it to do so, but it would be a cygwin thing, not
execnet specificly =/

Charles

On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau <matdrapeau at gmail.com> wrote:
> Hi,
> I am trying to connect to a windows box and execute a program which requires
> the display using execnet. The windows box has cygwin installed where sshd
> is running.
> Here is a small example of what I am trying to achieve.
>
>
> # remote.py script which should open the windows calculator utility
> from subprocess import call
> if __name__ == '__channelexec__':
> ??? call(["calc"])
>
>
> #code to run locally
> import execnet, remote
> gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
> ch = gw.remote_exec (remote)
> ch.waitclose ()
>
>
> Looking on the screen of my windows box, I don't see any calculator program!
> Is it expected?
>
> Thanks,
> Mathieu
>
> _______________________________________________
> execnet-dev mailing list
> execnet-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/execnet-dev
>
>


From matdrapeau at gmail.com  Thu Jul 29 16:54:25 2010
From: matdrapeau at gmail.com (Mathieu Drapeau)
Date: Thu, 29 Jul 2010 10:54:25 -0400
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com>
Message-ID: <AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com>

Charles,
Like you mentionned the problem was with sshd which didn't have the option
to "interact with the desktop".
This option could be enabled in the windows services panel. Unfortunately,
this can only be done in Windows XP pre-SP! Since then Microsoft has
disabled this option for security reasons.

So, I decided to switch to a socket server. I did see the socket server
script you posted but I am not able to make it work.
I do install and start the server but I am not able to connect to it. Seems
it is not running because when I try to stop it right after, it says the
service has not been started.

Do you have an updated version of your script? Do you have an idea why the
script seems not to be running?

thanks,
Mathieu

2010/7/27 Charles Solar <charlessolar at gmail.com>

> I suspect that cygwin's sshd probably operates in non-interactive
> mode.  Check the process list on the computer, does calc.exe show up?
> If so, it just means that sshd is indeed non-interactive which means
> it will not spawn graphical windows for logged in users.  There is
> probably some way for it to do so, but it would be a cygwin thing, not
> execnet specificly =/
>
> Charles
>
> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau <matdrapeau at gmail.com>
> wrote:
> > Hi,
> > I am trying to connect to a windows box and execute a program which
> requires
> > the display using execnet. The windows box has cygwin installed where
> sshd
> > is running.
> > Here is a small example of what I am trying to achieve.
> >
> >
> > # remote.py script which should open the windows calculator utility
> > from subprocess import call
> > if __name__ == '__channelexec__':
> >     call(["calc"])
> >
> >
> > #code to run locally
> > import execnet, remote
> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
> > ch = gw.remote_exec (remote)
> > ch.waitclose ()
> >
> >
> > Looking on the screen of my windows box, I don't see any calculator
> program!
> > Is it expected?
> >
> > Thanks,
> > Mathieu
> >
> > _______________________________________________
> > execnet-dev mailing list
> > execnet-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/execnet-dev
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100729/5f68ade8/attachment.html>

From charlessolar at gmail.com  Thu Jul 29 19:45:36 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Thu, 29 Jul 2010 12:45:36 -0500
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com> 
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com> 
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com>
Message-ID: <AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com>

well, i would check the process list, see if pythonservice.exe is
running.  You may need to run it as admin if you are not.  Open
services.msc to make sure the service is actually installed ( named
'python service' creatively enough).
If all that is working, then make sure you are connecting to it as a
socket gateway, not a ssh gateway.

Another thing you may want to check, make sure you have the latest
version I posted which should have a class def for IOJoiner at the
top.  That version has a few improvements and should work better than
the first one i posted.
Also, there is a script in the execnet scripts folder called
socketservice.py or something which works similarly and might work
better for you.

Charles

On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau <matdrapeau at gmail.com> wrote:
> Charles,
> Like you mentionned the problem was with sshd which didn't have the option
> to "interact with the desktop".
> This option could be enabled in the windows services panel. Unfortunately,
> this can only be done in Windows XP pre-SP! Since then Microsoft has
> disabled this option for security reasons.
>
> So, I decided to switch to a socket server. I did see the socket server
> script you posted but I am not able to make it work.
> I do install and start the server but I am not able to connect to it. Seems
> it is not running because when I try to stop it right after, it says the
> service has not been started.
>
> Do you have an updated version of your script? Do you have an idea why the
> script seems not to be running?
>
> thanks,
> Mathieu
>
> 2010/7/27 Charles Solar <charlessolar at gmail.com>
>>
>> I suspect that cygwin's sshd probably operates in non-interactive
>> mode. ?Check the process list on the computer, does calc.exe show up?
>> If so, it just means that sshd is indeed non-interactive which means
>> it will not spawn graphical windows for logged in users. ?There is
>> probably some way for it to do so, but it would be a cygwin thing, not
>> execnet specificly =/
>>
>> Charles
>>
>> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau <matdrapeau at gmail.com>
>> wrote:
>> > Hi,
>> > I am trying to connect to a windows box and execute a program which
>> > requires
>> > the display using execnet. The windows box has cygwin installed where
>> > sshd
>> > is running.
>> > Here is a small example of what I am trying to achieve.
>> >
>> >
>> > # remote.py script which should open the windows calculator utility
>> > from subprocess import call
>> > if __name__ == '__channelexec__':
>> > ??? call(["calc"])
>> >
>> >
>> > #code to run locally
>> > import execnet, remote
>> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
>> > ch = gw.remote_exec (remote)
>> > ch.waitclose ()
>> >
>> >
>> > Looking on the screen of my windows box, I don't see any calculator
>> > program!
>> > Is it expected?
>> >
>> > Thanks,
>> > Mathieu
>> >
>> > _______________________________________________
>> > execnet-dev mailing list
>> > execnet-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/execnet-dev
>> >
>> >
>
>


From matdrapeau at gmail.com  Thu Jul 29 20:37:48 2010
From: matdrapeau at gmail.com (Mathieu Drapeau)
Date: Thu, 29 Jul 2010 14:37:48 -0400
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com>
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com>
	<AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com>
Message-ID: <AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com>

Hi Charles,
I was able to install it finally, I needed to change the name of the file
(which was socketserver.py and causes conflict).

My problem now when I am trying to do a remote_exec, I am not able to debug
what's happening in the background. Do you have an idea where/how I can see
the trace?

thanks,
Mathieu


2010/7/29 Charles Solar <charlessolar at gmail.com>

> well, i would check the process list, see if pythonservice.exe is
> running.  You may need to run it as admin if you are not.  Open
> services.msc to make sure the service is actually installed ( named
> 'python service' creatively enough).
> If all that is working, then make sure you are connecting to it as a
> socket gateway, not a ssh gateway.
>
> Another thing you may want to check, make sure you have the latest
> version I posted which should have a class def for IOJoiner at the
> top.  That version has a few improvements and should work better than
> the first one i posted.
> Also, there is a script in the execnet scripts folder called
> socketservice.py or something which works similarly and might work
> better for you.
>
> Charles
>
> On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau <matdrapeau at gmail.com>
> wrote:
> > Charles,
> > Like you mentionned the problem was with sshd which didn't have the
> option
> > to "interact with the desktop".
> > This option could be enabled in the windows services panel.
> Unfortunately,
> > this can only be done in Windows XP pre-SP! Since then Microsoft has
> > disabled this option for security reasons.
> >
> > So, I decided to switch to a socket server. I did see the socket server
> > script you posted but I am not able to make it work.
> > I do install and start the server but I am not able to connect to it.
> Seems
> > it is not running because when I try to stop it right after, it says the
> > service has not been started.
> >
> > Do you have an updated version of your script? Do you have an idea why
> the
> > script seems not to be running?
> >
> > thanks,
> > Mathieu
> >
> > 2010/7/27 Charles Solar <charlessolar at gmail.com>
> >>
> >> I suspect that cygwin's sshd probably operates in non-interactive
> >> mode.  Check the process list on the computer, does calc.exe show up?
> >> If so, it just means that sshd is indeed non-interactive which means
> >> it will not spawn graphical windows for logged in users.  There is
> >> probably some way for it to do so, but it would be a cygwin thing, not
> >> execnet specificly =/
> >>
> >> Charles
> >>
> >> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau <matdrapeau at gmail.com>
> >> wrote:
> >> > Hi,
> >> > I am trying to connect to a windows box and execute a program which
> >> > requires
> >> > the display using execnet. The windows box has cygwin installed where
> >> > sshd
> >> > is running.
> >> > Here is a small example of what I am trying to achieve.
> >> >
> >> >
> >> > # remote.py script which should open the windows calculator utility
> >> > from subprocess import call
> >> > if __name__ == '__channelexec__':
> >> >     call(["calc"])
> >> >
> >> >
> >> > #code to run locally
> >> > import execnet, remote
> >> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
> >> > ch = gw.remote_exec (remote)
> >> > ch.waitclose ()
> >> >
> >> >
> >> > Looking on the screen of my windows box, I don't see any calculator
> >> > program!
> >> > Is it expected?
> >> >
> >> > Thanks,
> >> > Mathieu
> >> >
> >> > _______________________________________________
> >> > execnet-dev mailing list
> >> > execnet-dev at codespeak.net
> >> > http://codespeak.net/mailman/listinfo/execnet-dev
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100729/84c30e24/attachment.html>

From matdrapeau at gmail.com  Thu Jul 29 23:18:07 2010
From: matdrapeau at gmail.com (Mathieu Drapeau)
Date: Thu, 29 Jul 2010 17:18:07 -0400
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com>
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com>
	<AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com>
	<AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com>
Message-ID: <AANLkTi=ncA5oBt3xQ+sPo_OrQ7jZBe5ymJGoX-KbjwHs@mail.gmail.com>

Hi Charles,

In the source of your server script, I did change the makegateway definition
to this:

gw = execnet.makegateway("popen//pytho=C:\Python26\python.exe")

When I connect to the server, a python box pop and when I try to send basic
command to the server, I never receive any answers. The client just wait for
something.
Here is what I tried:
    ch = gw.remote_exec("import os; channel.send(os.getcwd())")
    ch.receive()

Do you know what could cause this problem?

Thanks,
Mathieu


2010/7/29 Mathieu Drapeau <matdrapeau at gmail.com>

> Hi Charles,
> I was able to install it finally, I needed to change the name of the file
> (which was socketserver.py and causes conflict).
>
> My problem now when I am trying to do a remote_exec, I am not able to debug
> what's happening in the background. Do you have an idea where/how I can see
> the trace?
>
> thanks,
> Mathieu
>
>
> 2010/7/29 Charles Solar <charlessolar at gmail.com>
>
> well, i would check the process list, see if pythonservice.exe is
>> running.  You may need to run it as admin if you are not.  Open
>> services.msc to make sure the service is actually installed ( named
>> 'python service' creatively enough).
>> If all that is working, then make sure you are connecting to it as a
>> socket gateway, not a ssh gateway.
>>
>> Another thing you may want to check, make sure you have the latest
>> version I posted which should have a class def for IOJoiner at the
>> top.  That version has a few improvements and should work better than
>> the first one i posted.
>> Also, there is a script in the execnet scripts folder called
>> socketservice.py or something which works similarly and might work
>> better for you.
>>
>> Charles
>>
>> On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau <matdrapeau at gmail.com>
>> wrote:
>> > Charles,
>> > Like you mentionned the problem was with sshd which didn't have the
>> option
>> > to "interact with the desktop".
>> > This option could be enabled in the windows services panel.
>> Unfortunately,
>> > this can only be done in Windows XP pre-SP! Since then Microsoft has
>> > disabled this option for security reasons.
>> >
>> > So, I decided to switch to a socket server. I did see the socket server
>> > script you posted but I am not able to make it work.
>> > I do install and start the server but I am not able to connect to it.
>> Seems
>> > it is not running because when I try to stop it right after, it says the
>> > service has not been started.
>> >
>> > Do you have an updated version of your script? Do you have an idea why
>> the
>> > script seems not to be running?
>> >
>> > thanks,
>> > Mathieu
>> >
>> > 2010/7/27 Charles Solar <charlessolar at gmail.com>
>> >>
>> >> I suspect that cygwin's sshd probably operates in non-interactive
>> >> mode.  Check the process list on the computer, does calc.exe show up?
>> >> If so, it just means that sshd is indeed non-interactive which means
>> >> it will not spawn graphical windows for logged in users.  There is
>> >> probably some way for it to do so, but it would be a cygwin thing, not
>> >> execnet specificly =/
>> >>
>> >> Charles
>> >>
>> >> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau <matdrapeau at gmail.com
>> >
>> >> wrote:
>> >> > Hi,
>> >> > I am trying to connect to a windows box and execute a program which
>> >> > requires
>> >> > the display using execnet. The windows box has cygwin installed where
>> >> > sshd
>> >> > is running.
>> >> > Here is a small example of what I am trying to achieve.
>> >> >
>> >> >
>> >> > # remote.py script which should open the windows calculator utility
>> >> > from subprocess import call
>> >> > if __name__ == '__channelexec__':
>> >> >     call(["calc"])
>> >> >
>> >> >
>> >> > #code to run locally
>> >> > import execnet, remote
>> >> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
>> >> > ch = gw.remote_exec (remote)
>> >> > ch.waitclose ()
>> >> >
>> >> >
>> >> > Looking on the screen of my windows box, I don't see any calculator
>> >> > program!
>> >> > Is it expected?
>> >> >
>> >> > Thanks,
>> >> > Mathieu
>> >> >
>> >> > _______________________________________________
>> >> > execnet-dev mailing list
>> >> > execnet-dev at codespeak.net
>> >> > http://codespeak.net/mailman/listinfo/execnet-dev
>> >> >
>> >> >
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100729/7db4d893/attachment.html>

From charlessolar at gmail.com  Thu Jul 29 23:42:59 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Thu, 29 Jul 2010 16:42:59 -0500
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTi=ncA5oBt3xQ+sPo_OrQ7jZBe5ymJGoX-KbjwHs@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com> 
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com> 
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com> 
	<AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com> 
	<AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com> 
	<AANLkTi=ncA5oBt3xQ+sPo_OrQ7jZBe5ymJGoX-KbjwHs@mail.gmail.com>
Message-ID: <AANLkTim6PzY2JwCLbKr2CwcBKET=si-BA1F0vrAU9YMQ@mail.gmail.com>

mm, well having a python window pop up is a bit suspicious.  Maybe the
popen io is not connecting correctly.
Maybe before trying the advanced process spawning service I wrote up,
try the basic 1 python instance service that is in execnet's script
directory.

Charles


On Thu, Jul 29, 2010 at 4:18 PM, Mathieu Drapeau <matdrapeau at gmail.com> wrote:
> Hi Charles,
>
> In the source of your server script, I did change the makegateway definition
> to this:
>
> gw = execnet.makegateway("popen//pytho=C:\Python26\python.exe")
>
> When I connect to the server, a python box pop and when I try to send basic
> command to the server, I never receive any answers. The client just wait for
> something.
> Here is what I tried:
> ??? ch = gw.remote_exec("import os; channel.send(os.getcwd())")
> ??? ch.receive()
>
> Do you know what could cause this problem?
>
> Thanks,
> Mathieu
>
>
> 2010/7/29 Mathieu Drapeau <matdrapeau at gmail.com>
>>
>> Hi Charles,
>> I was able to install it finally, I needed to change the name of the file
>> (which was socketserver.py and causes conflict).
>>
>> My problem now when I am trying to do a remote_exec, I am not able to
>> debug what's happening in the background. Do you have an idea where/how I
>> can see the trace?
>>
>> thanks,
>> Mathieu
>>
>>
>> 2010/7/29 Charles Solar <charlessolar at gmail.com>
>>>
>>> well, i would check the process list, see if pythonservice.exe is
>>> running. ?You may need to run it as admin if you are not. ?Open
>>> services.msc to make sure the service is actually installed ( named
>>> 'python service' creatively enough).
>>> If all that is working, then make sure you are connecting to it as a
>>> socket gateway, not a ssh gateway.
>>>
>>> Another thing you may want to check, make sure you have the latest
>>> version I posted which should have a class def for IOJoiner at the
>>> top. ?That version has a few improvements and should work better than
>>> the first one i posted.
>>> Also, there is a script in the execnet scripts folder called
>>> socketservice.py or something which works similarly and might work
>>> better for you.
>>>
>>> Charles
>>>
>>> On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau <matdrapeau at gmail.com>
>>> wrote:
>>> > Charles,
>>> > Like you mentionned the problem was with sshd which didn't have the
>>> > option
>>> > to "interact with the desktop".
>>> > This option could be enabled in the windows services panel.
>>> > Unfortunately,
>>> > this can only be done in Windows XP pre-SP! Since then Microsoft has
>>> > disabled this option for security reasons.
>>> >
>>> > So, I decided to switch to a socket server. I did see the socket server
>>> > script you posted but I am not able to make it work.
>>> > I do install and start the server but I am not able to connect to it.
>>> > Seems
>>> > it is not running because when I try to stop it right after, it says
>>> > the
>>> > service has not been started.
>>> >
>>> > Do you have an updated version of your script? Do you have an idea why
>>> > the
>>> > script seems not to be running?
>>> >
>>> > thanks,
>>> > Mathieu
>>> >
>>> > 2010/7/27 Charles Solar <charlessolar at gmail.com>
>>> >>
>>> >> I suspect that cygwin's sshd probably operates in non-interactive
>>> >> mode. ?Check the process list on the computer, does calc.exe show up?
>>> >> If so, it just means that sshd is indeed non-interactive which means
>>> >> it will not spawn graphical windows for logged in users. ?There is
>>> >> probably some way for it to do so, but it would be a cygwin thing, not
>>> >> execnet specificly =/
>>> >>
>>> >> Charles
>>> >>
>>> >> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau
>>> >> <matdrapeau at gmail.com>
>>> >> wrote:
>>> >> > Hi,
>>> >> > I am trying to connect to a windows box and execute a program which
>>> >> > requires
>>> >> > the display using execnet. The windows box has cygwin installed
>>> >> > where
>>> >> > sshd
>>> >> > is running.
>>> >> > Here is a small example of what I am trying to achieve.
>>> >> >
>>> >> >
>>> >> > # remote.py script which should open the windows calculator utility
>>> >> > from subprocess import call
>>> >> > if __name__ == '__channelexec__':
>>> >> > ??? call(["calc"])
>>> >> >
>>> >> >
>>> >> > #code to run locally
>>> >> > import execnet, remote
>>> >> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
>>> >> > ch = gw.remote_exec (remote)
>>> >> > ch.waitclose ()
>>> >> >
>>> >> >
>>> >> > Looking on the screen of my windows box, I don't see any calculator
>>> >> > program!
>>> >> > Is it expected?
>>> >> >
>>> >> > Thanks,
>>> >> > Mathieu
>>> >> >
>>> >> > _______________________________________________
>>> >> > execnet-dev mailing list
>>> >> > execnet-dev at codespeak.net
>>> >> > http://codespeak.net/mailman/listinfo/execnet-dev
>>> >> >
>>> >> >
>>> >
>>> >
>>
>
>


From matdrapeau at gmail.com  Fri Jul 30 01:36:26 2010
From: matdrapeau at gmail.com (Mathieu Drapeau)
Date: Thu, 29 Jul 2010 19:36:26 -0400
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTim6PzY2JwCLbKr2CwcBKET=si-BA1F0vrAU9YMQ@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com>
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com>
	<AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com>
	<AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com>
	<AANLkTi=ncA5oBt3xQ+sPo_OrQ7jZBe5ymJGoX-KbjwHs@mail.gmail.com>
	<AANLkTim6PzY2JwCLbKr2CwcBKET=si-BA1F0vrAU9YMQ@mail.gmail.com>
Message-ID: <AANLkTikXOw-2NWOkAWjf1fjeeB0==qeQapZtfc7ng_JQ@mail.gmail.com>

Yes I am currently using the other script and everything works fine.
Do you know how I can troubleshoot / debug your script?

Thanks,
Mathieu

2010/7/29 Charles Solar <charlessolar at gmail.com>

> mm, well having a python window pop up is a bit suspicious.  Maybe the
> popen io is not connecting correctly.
> Maybe before trying the advanced process spawning service I wrote up,
> try the basic 1 python instance service that is in execnet's script
> directory.
>
> Charles
>
>
> On Thu, Jul 29, 2010 at 4:18 PM, Mathieu Drapeau <matdrapeau at gmail.com>
> wrote:
> > Hi Charles,
> >
> > In the source of your server script, I did change the makegateway
> definition
> > to this:
> >
> > gw = execnet.makegateway("popen//pytho=C:\Python26\python.exe")
> >
> > When I connect to the server, a python box pop and when I try to send
> basic
> > command to the server, I never receive any answers. The client just wait
> for
> > something.
> > Here is what I tried:
> >     ch = gw.remote_exec("import os; channel.send(os.getcwd())")
> >     ch.receive()
> >
> > Do you know what could cause this problem?
> >
> > Thanks,
> > Mathieu
> >
> >
> > 2010/7/29 Mathieu Drapeau <matdrapeau at gmail.com>
> >>
> >> Hi Charles,
> >> I was able to install it finally, I needed to change the name of the
> file
> >> (which was socketserver.py and causes conflict).
> >>
> >> My problem now when I am trying to do a remote_exec, I am not able to
> >> debug what's happening in the background. Do you have an idea where/how
> I
> >> can see the trace?
> >>
> >> thanks,
> >> Mathieu
> >>
> >>
> >> 2010/7/29 Charles Solar <charlessolar at gmail.com>
> >>>
> >>> well, i would check the process list, see if pythonservice.exe is
> >>> running.  You may need to run it as admin if you are not.  Open
> >>> services.msc to make sure the service is actually installed ( named
> >>> 'python service' creatively enough).
> >>> If all that is working, then make sure you are connecting to it as a
> >>> socket gateway, not a ssh gateway.
> >>>
> >>> Another thing you may want to check, make sure you have the latest
> >>> version I posted which should have a class def for IOJoiner at the
> >>> top.  That version has a few improvements and should work better than
> >>> the first one i posted.
> >>> Also, there is a script in the execnet scripts folder called
> >>> socketservice.py or something which works similarly and might work
> >>> better for you.
> >>>
> >>> Charles
> >>>
> >>> On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau <matdrapeau at gmail.com
> >
> >>> wrote:
> >>> > Charles,
> >>> > Like you mentionned the problem was with sshd which didn't have the
> >>> > option
> >>> > to "interact with the desktop".
> >>> > This option could be enabled in the windows services panel.
> >>> > Unfortunately,
> >>> > this can only be done in Windows XP pre-SP! Since then Microsoft has
> >>> > disabled this option for security reasons.
> >>> >
> >>> > So, I decided to switch to a socket server. I did see the socket
> server
> >>> > script you posted but I am not able to make it work.
> >>> > I do install and start the server but I am not able to connect to it.
> >>> > Seems
> >>> > it is not running because when I try to stop it right after, it says
> >>> > the
> >>> > service has not been started.
> >>> >
> >>> > Do you have an updated version of your script? Do you have an idea
> why
> >>> > the
> >>> > script seems not to be running?
> >>> >
> >>> > thanks,
> >>> > Mathieu
> >>> >
> >>> > 2010/7/27 Charles Solar <charlessolar at gmail.com>
> >>> >>
> >>> >> I suspect that cygwin's sshd probably operates in non-interactive
> >>> >> mode.  Check the process list on the computer, does calc.exe show
> up?
> >>> >> If so, it just means that sshd is indeed non-interactive which means
> >>> >> it will not spawn graphical windows for logged in users.  There is
> >>> >> probably some way for it to do so, but it would be a cygwin thing,
> not
> >>> >> execnet specificly =/
> >>> >>
> >>> >> Charles
> >>> >>
> >>> >> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau
> >>> >> <matdrapeau at gmail.com>
> >>> >> wrote:
> >>> >> > Hi,
> >>> >> > I am trying to connect to a windows box and execute a program
> which
> >>> >> > requires
> >>> >> > the display using execnet. The windows box has cygwin installed
> >>> >> > where
> >>> >> > sshd
> >>> >> > is running.
> >>> >> > Here is a small example of what I am trying to achieve.
> >>> >> >
> >>> >> >
> >>> >> > # remote.py script which should open the windows calculator
> utility
> >>> >> > from subprocess import call
> >>> >> > if __name__ == '__channelexec__':
> >>> >> >     call(["calc"])
> >>> >> >
> >>> >> >
> >>> >> > #code to run locally
> >>> >> > import execnet, remote
> >>> >> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
> >>> >> > ch = gw.remote_exec (remote)
> >>> >> > ch.waitclose ()
> >>> >> >
> >>> >> >
> >>> >> > Looking on the screen of my windows box, I don't see any
> calculator
> >>> >> > program!
> >>> >> > Is it expected?
> >>> >> >
> >>> >> > Thanks,
> >>> >> > Mathieu
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > execnet-dev mailing list
> >>> >> > execnet-dev at codespeak.net
> >>> >> > http://codespeak.net/mailman/listinfo/execnet-dev
> >>> >> >
> >>> >> >
> >>> >
> >>> >
> >>
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100729/c349f8db/attachment.html>

From charlessolar at gmail.com  Fri Jul 30 01:50:07 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Thu, 29 Jul 2010 18:50:07 -0500
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTikXOw-2NWOkAWjf1fjeeB0==qeQapZtfc7ng_JQ@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com> 
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com> 
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com> 
	<AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com> 
	<AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com> 
	<AANLkTi=ncA5oBt3xQ+sPo_OrQ7jZBe5ymJGoX-KbjwHs@mail.gmail.com> 
	<AANLkTim6PzY2JwCLbKr2CwcBKET=si-BA1F0vrAU9YMQ@mail.gmail.com> 
	<AANLkTikXOw-2NWOkAWjf1fjeeB0==qeQapZtfc7ng_JQ@mail.gmail.com>
Message-ID: <AANLkTikgJm_dVwpuvwW7KV8PkYpbdyMqsA-EmNixeBbX@mail.gmail.com>

When I was debugging that script what I did was modify it so it was
not a service, but just a little daemon that ran in a console.  Then I
could put print commands in there and find out what was going on.
Something I think you should try is instead of adding the
"popen//python=", just add the python directory to your computers path
and remove that line.  I dont know, but maybe running makegateway like
that is opening a new python interactively, which would probably break
the popen io connection.
If that is all you changed then that is really all I can guess at what
the problem might be.  If you add some print statements and find out
what paths the service is taking when it fails I might be able to help
some more.

Charles

On Thu, Jul 29, 2010 at 6:36 PM, Mathieu Drapeau <matdrapeau at gmail.com> wrote:
> Yes I am currently using the other script and everything works fine.
> Do you know how I can troubleshoot / debug your script?
>
> Thanks,
> Mathieu
>
> 2010/7/29 Charles Solar <charlessolar at gmail.com>
>>
>> mm, well having a python window pop up is a bit suspicious. ?Maybe the
>> popen io is not connecting correctly.
>> Maybe before trying the advanced process spawning service I wrote up,
>> try the basic 1 python instance service that is in execnet's script
>> directory.
>>
>> Charles
>>
>>
>> On Thu, Jul 29, 2010 at 4:18 PM, Mathieu Drapeau <matdrapeau at gmail.com>
>> wrote:
>> > Hi Charles,
>> >
>> > In the source of your server script, I did change the makegateway
>> > definition
>> > to this:
>> >
>> > gw = execnet.makegateway("popen//pytho=C:\Python26\python.exe")
>> >
>> > When I connect to the server, a python box pop and when I try to send
>> > basic
>> > command to the server, I never receive any answers. The client just wait
>> > for
>> > something.
>> > Here is what I tried:
>> > ??? ch = gw.remote_exec("import os; channel.send(os.getcwd())")
>> > ??? ch.receive()
>> >
>> > Do you know what could cause this problem?
>> >
>> > Thanks,
>> > Mathieu
>> >
>> >
>> > 2010/7/29 Mathieu Drapeau <matdrapeau at gmail.com>
>> >>
>> >> Hi Charles,
>> >> I was able to install it finally, I needed to change the name of the
>> >> file
>> >> (which was socketserver.py and causes conflict).
>> >>
>> >> My problem now when I am trying to do a remote_exec, I am not able to
>> >> debug what's happening in the background. Do you have an idea where/how
>> >> I
>> >> can see the trace?
>> >>
>> >> thanks,
>> >> Mathieu
>> >>
>> >>
>> >> 2010/7/29 Charles Solar <charlessolar at gmail.com>
>> >>>
>> >>> well, i would check the process list, see if pythonservice.exe is
>> >>> running. ?You may need to run it as admin if you are not. ?Open
>> >>> services.msc to make sure the service is actually installed ( named
>> >>> 'python service' creatively enough).
>> >>> If all that is working, then make sure you are connecting to it as a
>> >>> socket gateway, not a ssh gateway.
>> >>>
>> >>> Another thing you may want to check, make sure you have the latest
>> >>> version I posted which should have a class def for IOJoiner at the
>> >>> top. ?That version has a few improvements and should work better than
>> >>> the first one i posted.
>> >>> Also, there is a script in the execnet scripts folder called
>> >>> socketservice.py or something which works similarly and might work
>> >>> better for you.
>> >>>
>> >>> Charles
>> >>>
>> >>> On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau
>> >>> <matdrapeau at gmail.com>
>> >>> wrote:
>> >>> > Charles,
>> >>> > Like you mentionned the problem was with sshd which didn't have the
>> >>> > option
>> >>> > to "interact with the desktop".
>> >>> > This option could be enabled in the windows services panel.
>> >>> > Unfortunately,
>> >>> > this can only be done in Windows XP pre-SP! Since then Microsoft has
>> >>> > disabled this option for security reasons.
>> >>> >
>> >>> > So, I decided to switch to a socket server. I did see the socket
>> >>> > server
>> >>> > script you posted but I am not able to make it work.
>> >>> > I do install and start the server but I am not able to connect to
>> >>> > it.
>> >>> > Seems
>> >>> > it is not running because when I try to stop it right after, it says
>> >>> > the
>> >>> > service has not been started.
>> >>> >
>> >>> > Do you have an updated version of your script? Do you have an idea
>> >>> > why
>> >>> > the
>> >>> > script seems not to be running?
>> >>> >
>> >>> > thanks,
>> >>> > Mathieu
>> >>> >
>> >>> > 2010/7/27 Charles Solar <charlessolar at gmail.com>
>> >>> >>
>> >>> >> I suspect that cygwin's sshd probably operates in non-interactive
>> >>> >> mode. ?Check the process list on the computer, does calc.exe show
>> >>> >> up?
>> >>> >> If so, it just means that sshd is indeed non-interactive which
>> >>> >> means
>> >>> >> it will not spawn graphical windows for logged in users. ?There is
>> >>> >> probably some way for it to do so, but it would be a cygwin thing,
>> >>> >> not
>> >>> >> execnet specificly =/
>> >>> >>
>> >>> >> Charles
>> >>> >>
>> >>> >> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau
>> >>> >> <matdrapeau at gmail.com>
>> >>> >> wrote:
>> >>> >> > Hi,
>> >>> >> > I am trying to connect to a windows box and execute a program
>> >>> >> > which
>> >>> >> > requires
>> >>> >> > the display using execnet. The windows box has cygwin installed
>> >>> >> > where
>> >>> >> > sshd
>> >>> >> > is running.
>> >>> >> > Here is a small example of what I am trying to achieve.
>> >>> >> >
>> >>> >> >
>> >>> >> > # remote.py script which should open the windows calculator
>> >>> >> > utility
>> >>> >> > from subprocess import call
>> >>> >> > if __name__ == '__channelexec__':
>> >>> >> > ??? call(["calc"])
>> >>> >> >
>> >>> >> >
>> >>> >> > #code to run locally
>> >>> >> > import execnet, remote
>> >>> >> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
>> >>> >> > ch = gw.remote_exec (remote)
>> >>> >> > ch.waitclose ()
>> >>> >> >
>> >>> >> >
>> >>> >> > Looking on the screen of my windows box, I don't see any
>> >>> >> > calculator
>> >>> >> > program!
>> >>> >> > Is it expected?
>> >>> >> >
>> >>> >> > Thanks,
>> >>> >> > Mathieu
>> >>> >> >
>> >>> >> > _______________________________________________
>> >>> >> > execnet-dev mailing list
>> >>> >> > execnet-dev at codespeak.net
>> >>> >> > http://codespeak.net/mailman/listinfo/execnet-dev
>> >>> >> >
>> >>> >> >
>> >>> >
>> >>> >
>> >>
>> >
>> >
>
>


From matdrapeau at gmail.com  Fri Jul 30 18:07:28 2010
From: matdrapeau at gmail.com (Mathieu Drapeau)
Date: Fri, 30 Jul 2010 12:07:28 -0400
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTikgJm_dVwpuvwW7KV8PkYpbdyMqsA-EmNixeBbX@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com>
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com>
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com>
	<AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com>
	<AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com>
	<AANLkTi=ncA5oBt3xQ+sPo_OrQ7jZBe5ymJGoX-KbjwHs@mail.gmail.com>
	<AANLkTim6PzY2JwCLbKr2CwcBKET=si-BA1F0vrAU9YMQ@mail.gmail.com>
	<AANLkTikXOw-2NWOkAWjf1fjeeB0==qeQapZtfc7ng_JQ@mail.gmail.com>
	<AANLkTikgJm_dVwpuvwW7KV8PkYpbdyMqsA-EmNixeBbX@mail.gmail.com>
Message-ID: <AANLkTim+17+nnv2_fS1sKV1k_DntVjhmxW9=qswVr1UC@mail.gmail.com>

Hi Charles,
I don't think it is the makegateway call that causes problems, I did try
bunch of different configuration without any help...
But the "source" variable seems weird. The way I understand it, it should be
a string and I should be able to output in the logs but everytime I try to
print its value, I get an error like this:

"RegisterEventSource/ReportEvent", the parameter is incorrect

Here is how I am trying to output the value of source:

try:
    servicemanager.LogErrorMsg("source = %s" % source)
except Exception as e:
    servicemanager.LogErrorMsg("error = %s" % e)

Any idea?

Thanks,
Mathieu

2010/7/29 Charles Solar <charlessolar at gmail.com>

> When I was debugging that script what I did was modify it so it was
> not a service, but just a little daemon that ran in a console.  Then I
> could put print commands in there and find out what was going on.
> Something I think you should try is instead of adding the
> "popen//python=", just add the python directory to your computers path
> and remove that line.  I dont know, but maybe running makegateway like
> that is opening a new python interactively, which would probably break
> the popen io connection.
> If that is all you changed then that is really all I can guess at what
> the problem might be.  If you add some print statements and find out
> what paths the service is taking when it fails I might be able to help
> some more.
>
> Charles
>
> On Thu, Jul 29, 2010 at 6:36 PM, Mathieu Drapeau <matdrapeau at gmail.com>
> wrote:
> > Yes I am currently using the other script and everything works fine.
> > Do you know how I can troubleshoot / debug your script?
> >
> > Thanks,
> > Mathieu
> >
> > 2010/7/29 Charles Solar <charlessolar at gmail.com>
> >>
> >> mm, well having a python window pop up is a bit suspicious.  Maybe the
> >> popen io is not connecting correctly.
> >> Maybe before trying the advanced process spawning service I wrote up,
> >> try the basic 1 python instance service that is in execnet's script
> >> directory.
> >>
> >> Charles
> >>
> >>
> >> On Thu, Jul 29, 2010 at 4:18 PM, Mathieu Drapeau <matdrapeau at gmail.com>
> >> wrote:
> >> > Hi Charles,
> >> >
> >> > In the source of your server script, I did change the makegateway
> >> > definition
> >> > to this:
> >> >
> >> > gw = execnet.makegateway("popen//pytho=C:\Python26\python.exe")
> >> >
> >> > When I connect to the server, a python box pop and when I try to send
> >> > basic
> >> > command to the server, I never receive any answers. The client just
> wait
> >> > for
> >> > something.
> >> > Here is what I tried:
> >> >     ch = gw.remote_exec("import os; channel.send(os.getcwd())")
> >> >     ch.receive()
> >> >
> >> > Do you know what could cause this problem?
> >> >
> >> > Thanks,
> >> > Mathieu
> >> >
> >> >
> >> > 2010/7/29 Mathieu Drapeau <matdrapeau at gmail.com>
> >> >>
> >> >> Hi Charles,
> >> >> I was able to install it finally, I needed to change the name of the
> >> >> file
> >> >> (which was socketserver.py and causes conflict).
> >> >>
> >> >> My problem now when I am trying to do a remote_exec, I am not able to
> >> >> debug what's happening in the background. Do you have an idea
> where/how
> >> >> I
> >> >> can see the trace?
> >> >>
> >> >> thanks,
> >> >> Mathieu
> >> >>
> >> >>
> >> >> 2010/7/29 Charles Solar <charlessolar at gmail.com>
> >> >>>
> >> >>> well, i would check the process list, see if pythonservice.exe is
> >> >>> running.  You may need to run it as admin if you are not.  Open
> >> >>> services.msc to make sure the service is actually installed ( named
> >> >>> 'python service' creatively enough).
> >> >>> If all that is working, then make sure you are connecting to it as a
> >> >>> socket gateway, not a ssh gateway.
> >> >>>
> >> >>> Another thing you may want to check, make sure you have the latest
> >> >>> version I posted which should have a class def for IOJoiner at the
> >> >>> top.  That version has a few improvements and should work better
> than
> >> >>> the first one i posted.
> >> >>> Also, there is a script in the execnet scripts folder called
> >> >>> socketservice.py or something which works similarly and might work
> >> >>> better for you.
> >> >>>
> >> >>> Charles
> >> >>>
> >> >>> On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau
> >> >>> <matdrapeau at gmail.com>
> >> >>> wrote:
> >> >>> > Charles,
> >> >>> > Like you mentionned the problem was with sshd which didn't have
> the
> >> >>> > option
> >> >>> > to "interact with the desktop".
> >> >>> > This option could be enabled in the windows services panel.
> >> >>> > Unfortunately,
> >> >>> > this can only be done in Windows XP pre-SP! Since then Microsoft
> has
> >> >>> > disabled this option for security reasons.
> >> >>> >
> >> >>> > So, I decided to switch to a socket server. I did see the socket
> >> >>> > server
> >> >>> > script you posted but I am not able to make it work.
> >> >>> > I do install and start the server but I am not able to connect to
> >> >>> > it.
> >> >>> > Seems
> >> >>> > it is not running because when I try to stop it right after, it
> says
> >> >>> > the
> >> >>> > service has not been started.
> >> >>> >
> >> >>> > Do you have an updated version of your script? Do you have an idea
> >> >>> > why
> >> >>> > the
> >> >>> > script seems not to be running?
> >> >>> >
> >> >>> > thanks,
> >> >>> > Mathieu
> >> >>> >
> >> >>> > 2010/7/27 Charles Solar <charlessolar at gmail.com>
> >> >>> >>
> >> >>> >> I suspect that cygwin's sshd probably operates in non-interactive
> >> >>> >> mode.  Check the process list on the computer, does calc.exe show
> >> >>> >> up?
> >> >>> >> If so, it just means that sshd is indeed non-interactive which
> >> >>> >> means
> >> >>> >> it will not spawn graphical windows for logged in users.  There
> is
> >> >>> >> probably some way for it to do so, but it would be a cygwin
> thing,
> >> >>> >> not
> >> >>> >> execnet specificly =/
> >> >>> >>
> >> >>> >> Charles
> >> >>> >>
> >> >>> >> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau
> >> >>> >> <matdrapeau at gmail.com>
> >> >>> >> wrote:
> >> >>> >> > Hi,
> >> >>> >> > I am trying to connect to a windows box and execute a program
> >> >>> >> > which
> >> >>> >> > requires
> >> >>> >> > the display using execnet. The windows box has cygwin installed
> >> >>> >> > where
> >> >>> >> > sshd
> >> >>> >> > is running.
> >> >>> >> > Here is a small example of what I am trying to achieve.
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > # remote.py script which should open the windows calculator
> >> >>> >> > utility
> >> >>> >> > from subprocess import call
> >> >>> >> > if __name__ == '__channelexec__':
> >> >>> >> >     call(["calc"])
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > #code to run locally
> >> >>> >> > import execnet, remote
> >> >>> >> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
> >> >>> >> > ch = gw.remote_exec (remote)
> >> >>> >> > ch.waitclose ()
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > Looking on the screen of my windows box, I don't see any
> >> >>> >> > calculator
> >> >>> >> > program!
> >> >>> >> > Is it expected?
> >> >>> >> >
> >> >>> >> > Thanks,
> >> >>> >> > Mathieu
> >> >>> >> >
> >> >>> >> > _______________________________________________
> >> >>> >> > execnet-dev mailing list
> >> >>> >> > execnet-dev at codespeak.net
> >> >>> >> > http://codespeak.net/mailman/listinfo/execnet-dev
> >> >>> >> >
> >> >>> >> >
> >> >>> >
> >> >>> >
> >> >>
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/execnet-dev/attachments/20100730/565b1dfa/attachment.html>

From holger at merlinux.eu  Sat Jul 31 00:34:04 2010
From: holger at merlinux.eu (holger krekel)
Date: Sat, 31 Jul 2010 00:34:04 +0200
Subject: [execnet-dev] execnet-1.0.8 and a green experiment
Message-ID: <20100730223404.GH1914@trillke.net>


I just released execnet-1.0.8, see the Changelog for details, below.

What is execnet? 
---------------------

execnet is a small pure python library that manages distributed code execution
across Python versions and host boundaries.  It provides a self-bootstrapping
share-nothing model with channels to send/receive data messages. It also 
helps with managing a cluster of (local or remote) processes and comes 
with tons of examples and API docs: 

    http://codespeak.net/execnet

green version of execnet 
--------------------------------

For those interested i hacked up a green version of execnet
which is API-compatible (and less stable) than the "main"
version.  It requires the eventlet library and is thus not fully 
self-bootstrapping (you need to install eventlet on all remote nodes
you want to execute code on).  There is no release but you can peruse

   http://code.google.com/p/execnet-green/source/checkout  

and "python setup.py develop" to play with it.  It will never spawn 
any operating system threads.  Rather remote_exec() will spawn 
greenlets/micro-threads of which you can easily have a few thousand.  
It probably only is stable enough for simple scenarios at this point. 

best,
holger

Changelog 1.0.8 (compared to 1.0.7):

- new ``gateway.remote_exec(func, **kwargs)`` style fo executing
  a pure function with parameters.  The function on the remote
  side also needs to accept a ``channel`` which allows it to
  communicate back and forth.  Thanks to Ronny Pfannschmidt
  for implementing it with special kudos to Maciej Fijalkowski
  for writing a "pure-function" checker so that on Python2.6
  onwards non-pure functions will be rejected.

- enhance rsyncing to also sync permissions (stat().st_mode) 
  of directories and files. 
  (should also resolve http://bitbucket.org/hpk42/py-trunk/issue/68/)

- fix rsyncing of symlinks, thanks to Charles Solar 
  (should also resolve http://bitbucket.org/hpk42/py-trunk/issue/70/)

- update internal usage of apipkg to 1.0b6

- remote_exec(module) now makes sure that the linecache is updated
  before reading and sending the source.  thanks Ronny, Matt. 

- removed all trailing whitespace from source files


From charlessolar at gmail.com  Sat Jul 31 21:22:43 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Sat, 31 Jul 2010 14:22:43 -0500
Subject: [execnet-dev] how to remotely open a program on win box?
In-Reply-To: <AANLkTim+17+nnv2_fS1sKV1k_DntVjhmxW9=qswVr1UC@mail.gmail.com>
References: <AANLkTinMDtns6Ta46_ZLKKHtbTMNRa_ue6HZSwbSeoOY@mail.gmail.com> 
	<AANLkTimr=Fc0KgcthpX-TYh162t21FY15X8FSyvkzbJ7@mail.gmail.com> 
	<AANLkTin=rHc5HooJ638+FvhbP8FDAYcoPV8d+wNu7AE7@mail.gmail.com> 
	<AANLkTimVKELwEpo7tRtfwc44eYiGfiO-wkAaUpwruUie@mail.gmail.com> 
	<AANLkTinE8ReX_6Y7eyjRQV6Dp0NuZdvz3cbKk9_qEqRW@mail.gmail.com> 
	<AANLkTi=ncA5oBt3xQ+sPo_OrQ7jZBe5ymJGoX-KbjwHs@mail.gmail.com> 
	<AANLkTim6PzY2JwCLbKr2CwcBKET=si-BA1F0vrAU9YMQ@mail.gmail.com> 
	<AANLkTikXOw-2NWOkAWjf1fjeeB0==qeQapZtfc7ng_JQ@mail.gmail.com> 
	<AANLkTikgJm_dVwpuvwW7KV8PkYpbdyMqsA-EmNixeBbX@mail.gmail.com> 
	<AANLkTim+17+nnv2_fS1sKV1k_DntVjhmxW9=qswVr1UC@mail.gmail.com>
Message-ID: <AANLkTi=Sj1Z+SpYbJPjLoZ+rc=xfaza_MbGH_NkyaiN7@mail.gmail.com>

I only vaguely remember trying to use the event manager as a debugging
tools with little to no success.  I would strongly recommend debugging
in a console window instead of a service instance.
Source is a string, but printing it out will not help you very much,
it just contains all the gateway bootstrapping code.  Actually, I bet
you are getting that error because the source string is to large.

To remove the windows service part from that file, what you need to do
is comment out all the __init__ code in the service class, and under
if __name__ == "-_main__" replace process args with something like
if __name__ == "__main__":
     svr=ServiceClass( None )
     svr.main()
    while True:
        time.sleep(0.1)

then you can put print statements where ever you want and they will
print to the console window you started the daemon in.

On Fri, Jul 30, 2010 at 11:07 AM, Mathieu Drapeau <matdrapeau at gmail.com> wrote:
> Hi Charles,
> I don't think it is the makegateway call that causes problems, I did try
> bunch of different configuration without any help...
> But the "source" variable seems weird. The way I understand it, it should be
> a string and I should be able to output in the logs but everytime I try to
> print its value, I get an error like this:
>
> "RegisterEventSource/ReportEvent", the parameter is incorrect
>
> Here is how I am trying to output the value of source:
>
> try:
> ??? servicemanager.LogErrorMsg("source = %s" % source)
> except Exception as e:
> ??? servicemanager.LogErrorMsg("error = %s" % e)
>
> Any idea?
>
> Thanks,
> Mathieu
>
> 2010/7/29 Charles Solar <charlessolar at gmail.com>
>>
>> When I was debugging that script what I did was modify it so it was
>> not a service, but just a little daemon that ran in a console. ?Then I
>> could put print commands in there and find out what was going on.
>> Something I think you should try is instead of adding the
>> "popen//python=", just add the python directory to your computers path
>> and remove that line. ?I dont know, but maybe running makegateway like
>> that is opening a new python interactively, which would probably break
>> the popen io connection.
>> If that is all you changed then that is really all I can guess at what
>> the problem might be. ?If you add some print statements and find out
>> what paths the service is taking when it fails I might be able to help
>> some more.
>>
>> Charles
>>
>> On Thu, Jul 29, 2010 at 6:36 PM, Mathieu Drapeau <matdrapeau at gmail.com>
>> wrote:
>> > Yes I am currently using the other script and everything works fine.
>> > Do you know how I can troubleshoot / debug your script?
>> >
>> > Thanks,
>> > Mathieu
>> >
>> > 2010/7/29 Charles Solar <charlessolar at gmail.com>
>> >>
>> >> mm, well having a python window pop up is a bit suspicious. ?Maybe the
>> >> popen io is not connecting correctly.
>> >> Maybe before trying the advanced process spawning service I wrote up,
>> >> try the basic 1 python instance service that is in execnet's script
>> >> directory.
>> >>
>> >> Charles
>> >>
>> >>
>> >> On Thu, Jul 29, 2010 at 4:18 PM, Mathieu Drapeau <matdrapeau at gmail.com>
>> >> wrote:
>> >> > Hi Charles,
>> >> >
>> >> > In the source of your server script, I did change the makegateway
>> >> > definition
>> >> > to this:
>> >> >
>> >> > gw = execnet.makegateway("popen//pytho=C:\Python26\python.exe")
>> >> >
>> >> > When I connect to the server, a python box pop and when I try to send
>> >> > basic
>> >> > command to the server, I never receive any answers. The client just
>> >> > wait
>> >> > for
>> >> > something.
>> >> > Here is what I tried:
>> >> > ??? ch = gw.remote_exec("import os; channel.send(os.getcwd())")
>> >> > ??? ch.receive()
>> >> >
>> >> > Do you know what could cause this problem?
>> >> >
>> >> > Thanks,
>> >> > Mathieu
>> >> >
>> >> >
>> >> > 2010/7/29 Mathieu Drapeau <matdrapeau at gmail.com>
>> >> >>
>> >> >> Hi Charles,
>> >> >> I was able to install it finally, I needed to change the name of the
>> >> >> file
>> >> >> (which was socketserver.py and causes conflict).
>> >> >>
>> >> >> My problem now when I am trying to do a remote_exec, I am not able
>> >> >> to
>> >> >> debug what's happening in the background. Do you have an idea
>> >> >> where/how
>> >> >> I
>> >> >> can see the trace?
>> >> >>
>> >> >> thanks,
>> >> >> Mathieu
>> >> >>
>> >> >>
>> >> >> 2010/7/29 Charles Solar <charlessolar at gmail.com>
>> >> >>>
>> >> >>> well, i would check the process list, see if pythonservice.exe is
>> >> >>> running. ?You may need to run it as admin if you are not. ?Open
>> >> >>> services.msc to make sure the service is actually installed ( named
>> >> >>> 'python service' creatively enough).
>> >> >>> If all that is working, then make sure you are connecting to it as
>> >> >>> a
>> >> >>> socket gateway, not a ssh gateway.
>> >> >>>
>> >> >>> Another thing you may want to check, make sure you have the latest
>> >> >>> version I posted which should have a class def for IOJoiner at the
>> >> >>> top. ?That version has a few improvements and should work better
>> >> >>> than
>> >> >>> the first one i posted.
>> >> >>> Also, there is a script in the execnet scripts folder called
>> >> >>> socketservice.py or something which works similarly and might work
>> >> >>> better for you.
>> >> >>>
>> >> >>> Charles
>> >> >>>
>> >> >>> On Thu, Jul 29, 2010 at 9:54 AM, Mathieu Drapeau
>> >> >>> <matdrapeau at gmail.com>
>> >> >>> wrote:
>> >> >>> > Charles,
>> >> >>> > Like you mentionned the problem was with sshd which didn't have
>> >> >>> > the
>> >> >>> > option
>> >> >>> > to "interact with the desktop".
>> >> >>> > This option could be enabled in the windows services panel.
>> >> >>> > Unfortunately,
>> >> >>> > this can only be done in Windows XP pre-SP! Since then Microsoft
>> >> >>> > has
>> >> >>> > disabled this option for security reasons.
>> >> >>> >
>> >> >>> > So, I decided to switch to a socket server. I did see the socket
>> >> >>> > server
>> >> >>> > script you posted but I am not able to make it work.
>> >> >>> > I do install and start the server but I am not able to connect to
>> >> >>> > it.
>> >> >>> > Seems
>> >> >>> > it is not running because when I try to stop it right after, it
>> >> >>> > says
>> >> >>> > the
>> >> >>> > service has not been started.
>> >> >>> >
>> >> >>> > Do you have an updated version of your script? Do you have an
>> >> >>> > idea
>> >> >>> > why
>> >> >>> > the
>> >> >>> > script seems not to be running?
>> >> >>> >
>> >> >>> > thanks,
>> >> >>> > Mathieu
>> >> >>> >
>> >> >>> > 2010/7/27 Charles Solar <charlessolar at gmail.com>
>> >> >>> >>
>> >> >>> >> I suspect that cygwin's sshd probably operates in
>> >> >>> >> non-interactive
>> >> >>> >> mode. ?Check the process list on the computer, does calc.exe
>> >> >>> >> show
>> >> >>> >> up?
>> >> >>> >> If so, it just means that sshd is indeed non-interactive which
>> >> >>> >> means
>> >> >>> >> it will not spawn graphical windows for logged in users. ?There
>> >> >>> >> is
>> >> >>> >> probably some way for it to do so, but it would be a cygwin
>> >> >>> >> thing,
>> >> >>> >> not
>> >> >>> >> execnet specificly =/
>> >> >>> >>
>> >> >>> >> Charles
>> >> >>> >>
>> >> >>> >> On Tue, Jul 27, 2010 at 4:36 PM, Mathieu Drapeau
>> >> >>> >> <matdrapeau at gmail.com>
>> >> >>> >> wrote:
>> >> >>> >> > Hi,
>> >> >>> >> > I am trying to connect to a windows box and execute a program
>> >> >>> >> > which
>> >> >>> >> > requires
>> >> >>> >> > the display using execnet. The windows box has cygwin
>> >> >>> >> > installed
>> >> >>> >> > where
>> >> >>> >> > sshd
>> >> >>> >> > is running.
>> >> >>> >> > Here is a small example of what I am trying to achieve.
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > # remote.py script which should open the windows calculator
>> >> >>> >> > utility
>> >> >>> >> > from subprocess import call
>> >> >>> >> > if __name__ == '__channelexec__':
>> >> >>> >> > ??? call(["calc"])
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > #code to run locally
>> >> >>> >> > import execnet, remote
>> >> >>> >> > gw = execnet.makegateway("ssh=%s" % WINBOX_IP)
>> >> >>> >> > ch = gw.remote_exec (remote)
>> >> >>> >> > ch.waitclose ()
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > Looking on the screen of my windows box, I don't see any
>> >> >>> >> > calculator
>> >> >>> >> > program!
>> >> >>> >> > Is it expected?
>> >> >>> >> >
>> >> >>> >> > Thanks,
>> >> >>> >> > Mathieu
>> >> >>> >> >
>> >> >>> >> > _______________________________________________
>> >> >>> >> > execnet-dev mailing list
>> >> >>> >> > execnet-dev at codespeak.net
>> >> >>> >> > http://codespeak.net/mailman/listinfo/execnet-dev
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >
>> >> >>> >
>> >> >>
>> >> >
>> >> >
>> >
>> >
>
>