From issues-reply at bitbucket.org Mon Jun 18 16:17:34 2012 From: issues-reply at bitbucket.org (Ronny Pfannschmidt) Date: Mon, 18 Jun 2012 14:17:34 -0000 Subject: [execnet-dev] [hpk42/execnet] test channel file interaction on different pythons (issue #7) Message-ID: <44c99098a90338a729f6c57581df4c8c@bitbucket.org> --- you can reply above this line --- New issue 7: test channel file interaction on different pythons https://bitbucket.org/hpk42/execnet/issue/7/test-channel-file-interaction-on-different Ronny Pfannschmidt: currently its not tested properly Responsible: RonnyPfannschmidt -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From issues-reply at bitbucket.org Mon Jun 18 21:44:48 2012 From: issues-reply at bitbucket.org (Ronny Pfannschmidt) Date: Mon, 18 Jun 2012 19:44:48 -0000 Subject: [execnet-dev] [hpk42/execnet] detailed checks of pid guessing for jython in relation to ssh gateways (issue #8) Message-ID: <48cff216ce37f5737c56e03e162fa36a@bitbucket.org> --- you can reply above this line --- New issue 8: detailed checks of pid guessing for jython in relation to ssh gateways https://bitbucket.org/hpk42/execnet/issue/8/detailed-checks-of-pid-guessing-for-jython Ronny Pfannschmidt: the code looks like it will use a remote pid for a local kill, thats bogous -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From holger at merlinux.eu Wed Jun 20 22:08:09 2012 From: holger at merlinux.eu (holger krekel) Date: Wed, 20 Jun 2012 20:08:09 +0000 Subject: [execnet-dev] execnet-1.1: cross-interpreter distributed execution library Message-ID: <20120620200809.GK11942@merlinux.eu> execnet-1.1 is a backward compatible beta release of the popular (>53000 pypi downloads of 1.0.9) cross-interpreter execution library. If you are in need of connecting Python2 and Python3 and/or want to throw PyPy in your deployment mix, then you might want to join Quora and many others and try out execnet. execnet provides a share-nothing model with channel-send/receive communication and distributed execution across many Python interpreters across version, platform and network barriers. See below for changes and see here for extensive documentation and tested examples: http://codespeak.net/execnet Particular thanks to Ronny Pfannschmidt for a lot of internal cleanups and to Alex Gaynor for providing feature patches. Have fun, holger 1.1 (compared to 1.0.9) -------------------------------- - introduce execnet.dumps/loads providing serialization between python interpreters, see http://codespeak.net/execnet/basics.html#dumps-loads - group.remote_exec now supports kwargs as well - support per channel string coercion configuration, helping with dealing with mixed Python2/Python3 environments. - Popen2IO.read now reads correct amounts of bytes from nonblocking fd's - added a ``dont_write_bytecode`` option to Popen gateways, this sets the ``sys.dont_write_bytecode`` flag on the spawned process, this only works on CPython 2.6 and higher. Thanks to Alex Gaynor. - added a pytest --broken-isp option to skip tests that assume DNS queries for unknown hosts actually are resolved as such (Thanks Alex Gaynor) - fix issue 1 - decouple string coercion of channels and gateway - fix issue #2 - properly reconfigure the channels string coercion for rsync, so it can send from python2 to python3 - refactor socketserver, so it can be directly remote_exec'd for starting a socket gateway on a remote From solrnps at gmail.com Thu Jun 21 22:58:16 2012 From: solrnps at gmail.com (solr nps) Date: Thu, 21 Jun 2012 13:58:16 -0700 Subject: [execnet-dev] remote process not terminating.. Message-ID: I am using exec net to run a java job on a remote server, the process runs fine and exits cleanly under normal circumstances, however when I hit Ctrl-C on the python process the remote java job keeps on running, the remote python dies in a few seconds. I am pasting all relevant code below, there are some self explanatory functions that I have left out. What am I doing wrong? I have read the documentation at http://codespeak.net/execnet/example/test_group.html#robust-termination-of-ssh-popen-processesand I think I am doing the same thing in my code. Thanks! **************** def poll_data(channel, host, output_file, service, threads): """ Function that is run remotely on the servers on a python interpretor, by virtue of the way we execve() the child process, if we go away, it goes away """ import subprocess java_location = '/home/user1/app/jdk/bin/java' jar_name = '/home/user1/data-poller.jar' command = [java_location, "-jar", jar_name, "-s", host, "-b", \ service, "-o", output_file, "-t", threads] channel.send("Running command: %s" % ' '.join(command)) p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) while True: retcode = p.poll() channel.send(p.stdout.readline()) if retcode is not None and retcode != 0: raise OSError('Unable to run child process: %s' % str(retcode)) def printData(data): """Simply prints anything it is given""" print data if __name__ == '__main__': group = execnet.Group() # Rsync can be used to move the jar to the right place, which makes keeping # up to date easier rsync = execnet.RSync('./libs') # Install signal handlers that will kill things if this process dies terminate = lambda: group.terminate(timeout=5.0) signal.signal(signal.SIGTERM, terminate) signal.signal(signal.SIGQUIT, terminate) signal.signal(signal.SIGHUP, terminate) servers,external_service,output_file,threads,port,endpoint = process_conf_file() try: channels = [] for server in servers: gw = group.makegateway('ssh=%s' %(server)) rsync.add_target(gw, '/tmp/libs') rsync.send() for gw, server in zip(group, servers): host = "http://%s:%s/%s" %(server, port, endpoint) channel = gw.remote_exec(poll_data, \ host = host, \ output_file = output_file, \ service = service, \ threads = threads) channel.setcallback(printData, endmarker=None) channels.append(channel) mch = execnet.MultiChannel(channels) mch.waitclose() except KeyboardInterrupt: terminate() -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Sat Jun 23 15:34:15 2012 From: holger at merlinux.eu (holger krekel) Date: Sat, 23 Jun 2012 13:34:15 +0000 Subject: [execnet-dev] remote process not terminating.. In-Reply-To: References: Message-ID: <20120623133415.GU11942@merlinux.eu> On Thu, Jun 21, 2012 at 13:58 -0700, solr nps wrote: > I am using exec net to run a java job on a remote server, the process runs > fine and exits cleanly under normal circumstances, however when I hit > Ctrl-C on the python process the remote java job keeps on running, the > remote python dies in a few seconds. I am pasting all relevant code below, > there are some self explanatory functions that I have left out. What am I > doing wrong? I have read the documentation at > http://codespeak.net/execnet/example/test_group.html#robust-termination-of-ssh-popen-processesand > I think I am doing the same thing in my code. > > Thanks! > > **************** > > def poll_data(channel, host, output_file, service, threads): > """ Function that is run remotely on the servers on a python > interpretor, by virtue of > the way we execve() the child process, if we go away, it goes away > """ > > import subprocess > > java_location = '/home/user1/app/jdk/bin/java' > jar_name = '/home/user1/data-poller.jar' > > command = [java_location, "-jar", jar_name, "-s", host, "-b", \ > service, "-o", output_file, "-t", threads] > > channel.send("Running command: %s" % ' '.join(command)) > p = subprocess.Popen(command, stdout=subprocess.PIPE, > stderr=subprocess.STDOUT) > > while True: > retcode = p.poll() > channel.send(p.stdout.readline()) > if retcode is not None and retcode != 0: > raise OSError('Unable to run child process: %s' % str(retcode)) > > def printData(data): > """Simply prints anything it is given""" > print data > > if __name__ == '__main__': > group = execnet.Group() > > # Rsync can be used to move the jar to the right place, which makes > keeping > # up to date easier > rsync = execnet.RSync('./libs') > > # Install signal handlers that will kill things if this process dies > terminate = lambda: group.terminate(timeout=5.0) > > signal.signal(signal.SIGTERM, terminate) > signal.signal(signal.SIGQUIT, terminate) > signal.signal(signal.SIGHUP, terminate) This should not be neccessary as groups terminate by default through python's atexit mechanism. Instead of trying to handle termination yourself - what traces do you see if you leave all termination code out and run your code with the environment variable EXECNET_DEBUG=2 (which will send traces to stderr) or EXECNET_DEBUG=1 (which will write traces to /tmp/execnet-debug-PID files)? To help me or others try things ourselves it would be great if you can strip down the (remaining) problem such that i only need Jython and Python and execnet to reproduce it. best & thanks, holger > servers,external_service,output_file,threads,port,endpoint = > process_conf_file() > try: > channels = [] > for server in servers: > gw = group.makegateway('ssh=%s' %(server)) > rsync.add_target(gw, '/tmp/libs') > > rsync.send() > > for gw, server in zip(group, servers): > host = "http://%s:%s/%s" %(server, port, endpoint) > channel = gw.remote_exec(poll_data, \ > host = host, \ > output_file = output_file, \ > service = service, \ > threads = threads) > > channel.setcallback(printData, endmarker=None) > channels.append(channel) > > mch = execnet.MultiChannel(channels) > mch.waitclose() > > except KeyboardInterrupt: > terminate() > _______________________________________________ > execnet-dev mailing list > execnet-dev at codespeak.net > http://codespeak.net/mailman/listinfo/execnet-dev