From kyle at bucebuce.com Wed Jun 9 22:48:03 2010 From: kyle at bucebuce.com (Kyle Jones) Date: Wed, 9 Jun 2010 13:48:03 -0700 Subject: [execnet-dev] Execnet and PyPy Sandbox Message-ID: Hello! I posted this in the IRC channel, but I thought I would throw it to the list for some advice. I am looking to get execnet working with a PyPy sandbox instance. Having tried it a couple of months ago with no success, I was curious if anyone has done this, or has some ideas of issues I would need to deal with to get it working? Changes I will need to make with execnet? Special ways I will need to deal with the interaction script? I know this isn't a PyPy list, but was hoping maybe someone had put some thought into it that knows that system a bit more. Thanks, Kyle -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlessolar at gmail.com Fri Jun 11 22:42:25 2010 From: charlessolar at gmail.com (Charles Solar) Date: Fri, 11 Jun 2010 15:42:25 -0500 Subject: [execnet-dev] Close remote gateway? Message-ID: I am attempting to use execnet to start remote python sessions for a testing framework I am working on. This framework will spawn a lot of remote gateways since I want each test to be run on its own process, a simpler and more clean approach imo. Things work great for the first few tests but I have been noticing that python interpreters are being left around. Well not really left around, but zombified and left for the OS to clean up. This is undesirable since in the course of several seconds this app could potentially spawn 20 or so python interpreters, each one only 'alive' for about half a second. I cannot wait around for the OS to kill the zombies. Is there someway I can remotely terminate a python interpreter through execnet? Currently, when the remote interpreter finishes his test I delete the gateway and channel instances from the host and I was hoping that would clean up the python instances on the remote server. But that does not seem to be the case. Thanks for any guidance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ronny.Pfannschmidt at gmx.de Sat Jun 12 14:54:28 2010 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sat, 12 Jun 2010 14:54:28 +0200 Subject: [execnet-dev] Close remote gateway? In-Reply-To: References: Message-ID: <1276347268.8061.4.camel@klappe2> On Fri, 2010-06-11 at 15:42 -0500, Charles Solar wrote: > I am attempting to use execnet to start remote python sessions for a > testing framework I am working on. This framework will spawn a lot of > remote gateways since I want each test to be run on its own process, a > simpler and more clean approach imo. > you might want to take a look into py.test's forked test execution (the --boxed param) and the pytest-xdist plugin for inspiration (or even a reason to contribute to pytest instead of making a own framework >:-) ) > Things work great for the first few tests but I have been noticing > that python interpreters are being left around. Well not really left > around, but zombified and left for the OS to clean up. This is > undesirable since in the course of several seconds this app could > potentially spawn 20 or so python interpreters, each one only 'alive' > for about half a second. I cannot wait around for the OS to kill the > zombies. > > Is there someway I can remotely terminate a python interpreter through > execnet? afair gateway.close() will clean up and should be used to ensure propper cleanup > > Currently, when the remote interpreter finishes his test I delete the > gateway and channel instances from the host and I was hoping that > would clean up the python instances on the remote server. But that > does not seem to be the case. > > Thanks for any guidance. -- ronny From holger at merlinux.eu Mon Jun 14 14:22:33 2010 From: holger at merlinux.eu (holger krekel) Date: Mon, 14 Jun 2010 14:22:33 +0200 Subject: [execnet-dev] Problem with execnet use In-Reply-To: References: Message-ID: <20100614122233.GI17693@trillke.net> Hi Pedro, On Sun, Jun 13, 2010 at 13:15 -0700, Pedro Marcal wrote: > Dear Holger, > You were kind enough to get me started. I tried to run the demo by Stuart > Perkins on NLTK. > I ran into the following problem. He suggested I contact the list. I did not > see a way to log in. You need to subscribe via http://codespeak.net/mailman/listinfo/execnet-dev after which you can post mails to execnet-dev at codespeak.net > Windows7 on ic5. "ic5"? i haven't used windows7 myself yet. > opening 4 gateways at localhost > > > Traceback (most recent call last): > File "C:/CD_data/run_tag_files.py", line 20, in > gw = execnet.makegateway('ssh=%s//nice=%d' % (host, NICE)) > File > "C:\Python26\lib\site-packages\execnet-1.0.6-py2.6.egg\execnet\multi.py", > line 80, in makegateway > ssh_config=spec.ssh_config, id=id) > File > "C:\Python26\lib\site-packages\execnet-1.0.6-py2.6.egg\execnet\gateway.py", > line 200, in __init__ > super(SshGateway, self).__init__(args, id=id) > File > "C:\Python26\lib\site-packages\execnet-1.0.6-py2.6.egg\execnet\gateway.py", > line 139, in __init__ > self._popen = p = Popen(args, stdin=PIPE, stdout=PIPE) > File "C:\Python26\lib\subprocess.py", line 633, in __init__ > errread, errwrite) > File "C:\Python26\lib\subprocess.py", line 842, in _execute_child > startupinfo) > WindowsError: [Error 2] The system cannot find the file specified > [DEBUG ON] Do you have ssh installed as a command line tool in your windows environment? I think people did so and had success using ssh-gateways on windows. If you really just want some "localhost" processes you could just use "popen//nice=..." which creates sub processes and works without external dependencies. cheers, holger From charlessolar at gmail.com Tue Jun 15 21:35:42 2010 From: charlessolar at gmail.com (Charles Solar) Date: Tue, 15 Jun 2010 14:35:42 -0500 Subject: [execnet-dev] Close remote gateway? In-Reply-To: <1276347268.8061.4.camel@klappe2> References: <1276347268.8061.4.camel@klappe2> Message-ID: Thanks for the pointers. execnet.gateway does not have a close method, but I assume you mean exit because it does have that. Unfortunately after testing it out the last couple of days this method does not do what I want either. It just orphans the remote processes and they eventually get cleaned up 5-10 seconds later by the os. As a result of further testing I am now planning on shifting my design away from many python interpreters in favor one interpreter per remote machine so this is not really a problem anymore for me. However it would still be nice to be able to call exit on a remote gateway and have it close immediately. Also I have been referring to pytest-xdist for some things, but the way I need to test my app requires a different sort of framework capable of handling tests that communicate with other machines. xdist only handles running unit tests on other machines in a closed environment. Thanks On Sat, Jun 12, 2010 at 7:54 AM, Ronny Pfannschmidt < Ronny.Pfannschmidt at gmx.de> wrote: > On Fri, 2010-06-11 at 15:42 -0500, Charles Solar wrote: > > I am attempting to use execnet to start remote python sessions for a > > testing framework I am working on. This framework will spawn a lot of > > remote gateways since I want each test to be run on its own process, a > > simpler and more clean approach imo. > > > you might want to take a look into py.test's forked test execution > (the --boxed param) and the pytest-xdist plugin for inspiration > (or even a reason to contribute to pytest instead of making a own > framework >:-) ) > > > Things work great for the first few tests but I have been noticing > > that python interpreters are being left around. Well not really left > > around, but zombified and left for the OS to clean up. This is > > undesirable since in the course of several seconds this app could > > potentially spawn 20 or so python interpreters, each one only 'alive' > > for about half a second. I cannot wait around for the OS to kill the > > zombies. > > > > Is there someway I can remotely terminate a python interpreter through > > execnet? > > afair gateway.close() will clean up and should be used to ensure propper > cleanup > > > > > Currently, when the remote interpreter finishes his test I delete the > > gateway and channel instances from the host and I was hoping that > > would clean up the python instances on the remote server. But that > > does not seem to be the case. > > > > Thanks for any guidance. > > -- ronny > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue Jun 15 21:59:18 2010 From: holger at merlinux.eu (holger krekel) Date: Tue, 15 Jun 2010 21:59:18 +0200 Subject: [execnet-dev] Close remote gateway? In-Reply-To: References: <1276347268.8061.4.camel@klappe2> Message-ID: <20100615195918.GQ17693@trillke.net> Hi Charles, i am interestd to resolve the issue you are seeing. i wonder if you looked into these examples: http://codespeak.net/execnet/example/test_group.html#robust-termination-of-ssh-popen-processes If this did not help your case, you could maybe change the "10.0" parameter in execnet/gateway_base.py:687 which tries to terminate execution. If this helps we could see to make this parameter API-changeable. cheers, holger On Tue, Jun 15, 2010 at 14:35 -0500, Charles Solar wrote: > Thanks for the pointers. execnet.gateway does not have a close method, but > I assume you mean exit because it does have that. > Unfortunately after testing it out the last couple of days this method does > not do what I want either. It just orphans the remote processes and they > eventually get cleaned up 5-10 seconds later by the os. > > > As a result of further testing I am now planning on shifting my design away > from many python interpreters in favor one interpreter per remote machine so > this is not really a problem anymore for me. However it would still be nice > to be able to call exit on a remote gateway and have it close immediately. > > Also I have been referring to pytest-xdist for some things, but the way I > need to test my app requires a different sort of framework capable of > handling tests that communicate with other machines. xdist only handles > running unit tests on other machines in a closed environment. > > Thanks > > On Sat, Jun 12, 2010 at 7:54 AM, Ronny Pfannschmidt < > Ronny.Pfannschmidt at gmx.de> wrote: > > > On Fri, 2010-06-11 at 15:42 -0500, Charles Solar wrote: > > > I am attempting to use execnet to start remote python sessions for a > > > testing framework I am working on. This framework will spawn a lot of > > > remote gateways since I want each test to be run on its own process, a > > > simpler and more clean approach imo. > > > > > you might want to take a look into py.test's forked test execution > > (the --boxed param) and the pytest-xdist plugin for inspiration > > (or even a reason to contribute to pytest instead of making a own > > framework >:-) ) > > > > > Things work great for the first few tests but I have been noticing > > > that python interpreters are being left around. Well not really left > > > around, but zombified and left for the OS to clean up. This is > > > undesirable since in the course of several seconds this app could > > > potentially spawn 20 or so python interpreters, each one only 'alive' > > > for about half a second. I cannot wait around for the OS to kill the > > > zombies. > > > > > > Is there someway I can remotely terminate a python interpreter through > > > execnet? > > > > afair gateway.close() will clean up and should be used to ensure propper > > cleanup > > > > > > > > Currently, when the remote interpreter finishes his test I delete the > > > gateway and channel instances from the host and I was hoping that > > > would clean up the python instances on the remote server. But that > > > does not seem to be the case. > > > > > > Thanks for any guidance. > > > > -- ronny > > > > > _______________________________________________ > execnet-dev mailing list > execnet-dev at codespeak.net > http://codespeak.net/mailman/listinfo/execnet-dev -- From charlessolar at gmail.com Tue Jun 15 22:18:34 2010 From: charlessolar at gmail.com (Charles Solar) Date: Tue, 15 Jun 2010 15:18:34 -0500 Subject: [execnet-dev] Close remote gateway? In-Reply-To: <20100615195918.GQ17693@trillke.net> References: <1276347268.8061.4.camel@klappe2> <20100615195918.GQ17693@trillke.net> Message-ID: I did see those examples and I remember that I had then wished that gateway.exit() had the timeout option that I could use. I would use gateway groups, but I am afraid its too restrictive to my design. Currently I setup a bunch of gateways to the target machines, about 10 or so, and hold onto the gateway instance in a list along with other params like hostname, unique id, etc. I then allow the user to close subsets of remote hosts so something like this works: server = RemoteServer() server.open( [ 'server1', 'server2', 'server3' ] ) server.close( ['server1','server2'] ) which would be much harder if I opened all the servers under one group. I went to the location you suggested and changed the value to 1.0 and it did in fact help. The instances fell down just 1 second after I called exit. So yes, being able to do gateway.exit( timeout=0.5 ) or something would be very nice. I guess my remote clients are doing busy work that is preventing the clean shutdown for 10 seconds, which would explain what I am seeing. Thanks On Tue, Jun 15, 2010 at 2:59 PM, holger krekel wrote: > Hi Charles, > > i am interestd to resolve the issue you are seeing. > i wonder if you looked into these examples: > > > http://codespeak.net/execnet/example/test_group.html#robust-termination-of-ssh-popen-processes > > If this did not help your case, you could maybe change the "10.0" parameter > in execnet/gateway_base.py:687 which tries to terminate execution. > If this helps we could see to make this parameter API-changeable. > > cheers, > holger > > On Tue, Jun 15, 2010 at 14:35 -0500, Charles Solar wrote: > > Thanks for the pointers. execnet.gateway does not have a close method, > but > > I assume you mean exit because it does have that. > > Unfortunately after testing it out the last couple of days this method > does > > not do what I want either. It just orphans the remote processes and they > > eventually get cleaned up 5-10 seconds later by the os. > > > > > > As a result of further testing I am now planning on shifting my design > away > > from many python interpreters in favor one interpreter per remote machine > so > > this is not really a problem anymore for me. However it would still be > nice > > to be able to call exit on a remote gateway and have it close > immediately. > > > > Also I have been referring to pytest-xdist for some things, but the way I > > need to test my app requires a different sort of framework capable of > > handling tests that communicate with other machines. xdist only handles > > running unit tests on other machines in a closed environment. > > > > Thanks > > > > On Sat, Jun 12, 2010 at 7:54 AM, Ronny Pfannschmidt < > > Ronny.Pfannschmidt at gmx.de> wrote: > > > > > On Fri, 2010-06-11 at 15:42 -0500, Charles Solar wrote: > > > > I am attempting to use execnet to start remote python sessions for a > > > > testing framework I am working on. This framework will spawn a lot > of > > > > remote gateways since I want each test to be run on its own process, > a > > > > simpler and more clean approach imo. > > > > > > > you might want to take a look into py.test's forked test execution > > > (the --boxed param) and the pytest-xdist plugin for inspiration > > > (or even a reason to contribute to pytest instead of making a own > > > framework >:-) ) > > > > > > > Things work great for the first few tests but I have been noticing > > > > that python interpreters are being left around. Well not really left > > > > around, but zombified and left for the OS to clean up. This is > > > > undesirable since in the course of several seconds this app could > > > > potentially spawn 20 or so python interpreters, each one only 'alive' > > > > for about half a second. I cannot wait around for the OS to kill the > > > > zombies. > > > > > > > > Is there someway I can remotely terminate a python interpreter > through > > > > execnet? > > > > > > afair gateway.close() will clean up and should be used to ensure > propper > > > cleanup > > > > > > > > > > > Currently, when the remote interpreter finishes his test I delete the > > > > gateway and channel instances from the host and I was hoping that > > > > would clean up the python instances on the remote server. But that > > > > does not seem to be the case. > > > > > > > > Thanks for any guidance. > > > > > > -- ronny > > > > > > > > > _______________________________________________ > > execnet-dev mailing list > > execnet-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/execnet-dev > > > -- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Wed Jun 16 14:03:05 2010 From: holger at merlinux.eu (holger krekel) Date: Wed, 16 Jun 2010 14:03:05 +0200 Subject: [execnet-dev] Close remote gateway? In-Reply-To: References: <1276347268.8061.4.camel@klappe2> <20100615195918.GQ17693@trillke.net> Message-ID: <20100616120305.GT17693@trillke.net> On Tue, Jun 15, 2010 at 15:18 -0500, Charles Solar wrote: > I did see those examples and I remember that I had then wished that > gateway.exit() had the timeout option that I could use. > I would use gateway groups, but I am afraid its too restrictive to my > design. Currently I setup a bunch of gateways to the target machines, about > 10 or so, and hold onto the gateway instance in a list along with other > params like hostname, unique id, etc. I then allow the user to close > subsets of remote hosts so something like this works: > > server = RemoteServer() > server.open( [ 'server1', 'server2', 'server3' ] ) > server.close( ['server1','server2'] ) > > which would be much harder if I opened all the servers under one group. Under the hood you could still use groups - eases wholesale termination at least. We could also refine the API to support partial exit. I'd be happy to accept patches that help your use case. > I went to the location you suggested and changed the value to 1.0 and it did > in fact help. The instances fell down just 1 second after I called exit. > So yes, being able to do gateway.exit( timeout=0.5 ) or something would be > very nice. I guess my remote clients are doing busy work that is preventing > the clean shutdown for 10 seconds, which would explain what I am seeing. We probably don't want gateway.exit() to wait until the remote side terminates. So someting like gateway.exit(killexec=10) makes sense. This would also become a parameter to group.terminate(killexec=10) or be derived as "timeout * 0.5" or so. Terminate accepts a timeout paramater already because it actually waits for termination - but for all hosts at once. Makes sense to you? best, holger From jgustak at gmail.com Thu Jun 24 14:30:00 2010 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 24 Jun 2010 13:30:00 +0100 Subject: [execnet-dev] adding env option to xspec Message-ID: Hello, I need to specify some ENV variable when running a sub-process (this case for xdist). AFAIK there is no support for such feature. I do a workaround by popen//python=./python_with_env where ./python_with_env does specify required ENV. It would be nice to have xspec env option (e.g. popen//env='VAR1=value VAR2=value' ). I can fry up a patch for this functionality if you consider it a feature worth inclusion. Cheers, Jakub Gustak From holger at merlinux.eu Thu Jun 24 14:53:05 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 24 Jun 2010 14:53:05 +0200 Subject: [execnet-dev] adding env option to xspec In-Reply-To: References: Message-ID: <20100624125304.GC14601@trillke.net> Hi Jakub, not sure it makes sense to add this to execnet, maybe better to xdist code? Things is it's easy to set environment variables when initializing gateways yourself. Just when using pytest-xdist you cannot easily do this because that API level is not exposed. So xdist could grow a configuration option to interpret some additional xspec settings. I suggest to use popen//env:NAME=value as syntax. That being said i am not totally opposed to adding it directly to execnet. Maybe other people here on the list also have comments ... best, holger On Thu, Jun 24, 2010 at 13:30 +0100, Jakub Gustak wrote: > Hello, > > I need to specify some ENV variable when running a sub-process (this > case for xdist). AFAIK there is no support for such feature. > > I do a workaround by popen//python=./python_with_env where > ./python_with_env does specify required ENV. > > It would be nice to have xspec env option (e.g. popen//env='VAR1=value > VAR2=value' ). > > I can fry up a patch for this functionality if you consider it a > feature worth inclusion. > > Cheers, > Jakub Gustak > _______________________________________________ > execnet-dev mailing list > execnet-dev at codespeak.net > http://codespeak.net/mailman/listinfo/execnet-dev > --