From opensource at ronnypfannschmidt.de Sat Feb 14 23:50:16 2015 From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Sat, 14 Feb 2015 23:50:16 +0100 Subject: [execnet-dev] [ANN] Execnet 1.3.0 released Message-ID: Hello, is my pleasure to announce the release of execnet 1.3.0 availiable at https://pypi.python.org/pypi/execnet/1.3.0 -- Ronny PS: execnet: distributed Python deployment and communication ======================================================== .. _execnet: http://codespeak.net/execnet execnet_ provides carefully tested means to ad-hoc interact with Python interpreters across version, platform and network barriers. It provides a minimal and fast API targetting the following uses: * distribute tasks to local or remote processes * write and deploy hybrid multi-process applications * write scripts to administer multiple hosts Features ------------------ * zero-install bootstrapping: no remote installation required! * flexible communication: send/receive as well as callback/queue mechanisms supported * simple serialization of python builtin types (no pickling) * grouped creation and robust termination of processes * well tested between CPython 2.6-3.X, Jython 2.5.1 and PyPy 2.2 interpreters. * interoperable between Windows and Unix-ish systems. * integrates with different threading models, including standard os threads, eventlet and gevent based systems. Changelog =========== 1.3.0 -------------------------------- - fix issue33: index.txt to correctly mention MIT instead of GPL. - fix issue35: adapt some doctests, fix some channel tests for py3. - use subprocess32 when available for python < 3. - try to be a bit more careful when interpreter is shutting down to avoid random exceptions, thanks Alfredo Deza. - ignore errors on orphan file removal when rsyncing - fix issue34: limit use of import based bootstrap 1.2 -------------------------------- - fix issue22 -- during interpreter shutdown don't throw an exception when we can't send a termination sequence anymore as we are about to die anyway. - fix issue24 -- allow concurrent creation of gateways by guarding automatic id creation by a look. Thanks tlecomte. - majorly refactor internal thread and IO handling. execnet can now operate on different thread models, defaults to "thread" but allows for eventlet and gevent if it is installed. - gateway.remote_exec() will now execute in multiple threads on the other side by default. The previous neccessity of running "gateway.remote_init_threads()" to allow for such concurrency is gone. The latter method is now a no-op and will be removed in future versions of execnet. - fix issue20: prevent AttributError at interpreter shutdown by not trying to send close/last_message messages if the world around is half destroyed. - fix issue21: allow to create local gateways with sudo aka makegateway("popen//python=sudo python"). Thanks Alfredo Deza for the PR. - streamline gateway termination and simplify proxy implementation. add more internal tracing. - if execution hangs in computation, we now try to send a SIGINT to ourselves on Unix platforms instead of just calling thread.interrupt_main() - change license from GPL to MIT - introduce execnet.dump/load variants of dumps/loads serializing/unserializing mechanism. - improve channel.receive() communication latency on python2 by changing the default timeout of the underlying Queue.get to a regular None instead of the previous default -1 which caused an internal positive timeout value (a hack probably introduced to allow CTRL-C to pass through for References: <544821B8.9020404@gmail.com> Message-ID: <0a7007c4-b6be-4a56-b3f1-9a5ea27ef409@ronnypfannschmidt.de> Recent execnet gatways can be bootstraped via another gateway so >>> group = execnet.Group() >>> a = group.makegateway('ssh=a//id=a') >>> b = group.makegateway('ssh=b//via=a') should do the trick On Wednesday, October 22, 2014 11:29:28 PM CEST, Prasanna Mulgaonkar wrote: > I am trying to use execnet for a monitoring application where > one server (say A) needs to reach into another (say C) and > execute some python code. This works just fine using execnet, > and the fact that it is a zero footprint mechanism means that > (other than python) C does not have to have anything else > preconfigured on it. > > Occasionally I see situations where there is no direct > connectivity from A to C, and one needs to hop through some > gateway device (say B), where B simply serves as a relay for any > channels coming from A or C. To do this, I send over a python > module from A to B which then opens another gateway to C. > > However, this solution requires that B have execnet installed. > > Is there any way that execnet can bootstrap itself onto the > intermediate machine B? I can open SSH to B, I have root on B if > I need to (which means I can install execnet if it does not > exist). But the ideal solution would be to send execnet itself > as a module, or at least the key parts of it over the channel > from A to B. > > Pointers much appreciated. > > --prasanna From prasanna.mulgaonkar at gmail.com Wed Feb 18 18:55:06 2015 From: prasanna.mulgaonkar at gmail.com (Prasanna Mulgaonkar) Date: Wed, 18 Feb 2015 09:55:06 -0800 Subject: [execnet-dev] Multihopping execnet connections In-Reply-To: <0a7007c4-b6be-4a56-b3f1-9a5ea27ef409@ronnypfannschmidt.de> References: <544821B8.9020404@gmail.com> <0a7007c4-b6be-4a56-b3f1-9a5ea27ef409@ronnypfannschmidt.de> Message-ID: <54E4D1FA.20400@gmail.com> Thanks. Willl try that. Did not see the via flag documented anywhere before. --prasanna On 02/18/2015 08:57 AM, Ronny Pfannschmidt wrote: > > Recent execnet gatways can be bootstraped via another gateway > > so >>>> group = execnet.Group() >>>> a = group.makegateway('ssh=a//id=a') >>>> b = group.makegateway('ssh=b//via=a') > > should do the trick > On Wednesday, October 22, 2014 11:29:28 PM CEST, Prasanna Mulgaonkar > wrote: >> I am trying to use execnet for a monitoring application where one >> server (say A) needs to reach into another (say C) and execute some >> python code. This works just fine using execnet, and the fact that >> it is a zero footprint mechanism means that (other than python) C >> does not have to have anything else preconfigured on it. >> >> Occasionally I see situations where there is no direct connectivity >> from A to C, and one needs to hop through some gateway device (say >> B), where B simply serves as a relay for any channels coming from A >> or C. To do this, I send over a python module from A to B which then >> opens another gateway to C. >> >> However, this solution requires that B have execnet installed. >> >> Is there any way that execnet can bootstrap itself onto the >> intermediate machine B? I can open SSH to B, I have root on B if I >> need to (which means I can install execnet if it does not exist). But >> the ideal solution would be to send execnet itself as a module, or at >> least the key parts of it over the channel from A to B. >> >> Pointers much appreciated. >> >> --prasanna