From issues-reply at bitbucket.org Wed Jan 22 16:51:22 2014 From: issues-reply at bitbucket.org (Crusader Ky) Date: Wed, 22 Jan 2014 15:51:22 -0000 Subject: [execnet-dev] Issue #28: Group.__init() should be parallelised (hpk42/execnet) Message-ID: <20140122155122.7355.61824@app18.ash-private.bitbucket.org> New issue 28: Group.__init() should be parallelised https://bitbucket.org/hpk42/execnet/issue/28/group__init-should-be-parallelised Crusader Ky: Consider the following snippet: hosts = [...list of hosts...] group = execnet.Group(("ssh %s" % host for host in hosts)) group.remote_exec(something) Imagine having a grid of 200 hosts. While Group.remote_exec() will run on all the hosts in parallel, Group.__init__() will connect via ssh to each one of them, *serially*. On a typical LAN, this easily accounts to *minutes*. I attach a patch for multi.py that is intended to fix the problem. However, with that patch the program deadlocks in gateway_base.py, line 360, and I can't figure out the reason. From holger at merlinux.eu Wed Jan 29 14:21:16 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 29 Jan 2014 13:21:16 +0000 Subject: [execnet-dev] execnet-1.2 released: gevent/eventlet support, many improvements Message-ID: <20140129132116.GF13547@merlinux.eu> Hi all, just released execnet-1.2, the tool for writing distributed zero-install python programs. The release contains lots of improvements and fixes. Most notably execnet now supports to use gevent/eventlet models on each of the initiating side and the remote side (configurable separately). See the changelog below for details. Install it via: pip install -U execnet Read the docs here: http://codespeak.net/execnet/ have fun playing, holger 1.2 -------------------------------- - fix issue22 -- during interpreter shutdown don't throw an exception when we can't send a termination sequence anymore as we are about to die anyway. - fix issue24 -- allow concurrent creation of gateways by guarding automatic id creation by a look. Thanks tlecomte. - majorly refactor internal thread and IO handling. execnet can now operate on different thread models, defaults to "thread" but allows for eventlet and gevent if it is installed. - gateway.remote_exec() will now execute in multiple threads on the other side by default. The previous neccessity of running "gateway.remote_init_threads()" to allow for such concurrency is gone. The latter method is now a no-op and will be removed in future versions of execnet. - fix issue20: prevent AttributError at interpreter shutdown by not trying to send close/last_message messages if the world around is half destroyed. - fix issue21: allow to create local gateways with sudo aka makegateway("popen//python=sudo python"). Thanks Alfredo Deza for the PR. - streamline gateway termination and simplify proxy implementation. add more internal tracing. - if execution hangs in computation, we now try to send a SIGINT to ourselves on Unix platforms instead of just calling thread.interrupt_main() - change license from GPL to MIT - introduce execnet.dump/load variants of dumps/loads serializing/unserializing mechanism. - improve channel.receive() communication latency on python2 by changing the default timeout of the underlying Queue.get to a regular None instead of the previous default -1 which caused an internal positive timeout value (a hack probably introduced to allow CTRL-C to pass through for New issue 29: new `TypeError` issues when closing the gateway https://bitbucket.org/hpk42/execnet/issue/29/new-typeerror-issues-when-closing-the Alfredo Deza: We are seeing new `TypeError` exceptions when the gateway. These are really hard to replicate but all too common now in the logs of our internal tests (that use the latest release of execnet, 1.2). ``` Unhandled exception in thread started by Traceback (most recent call last): File "/home/ubuntu/cephtest/ceph-deploy/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py", line 245, in run_and_release with self._running_lock: File "/usr/lib64/python2.6/threading.py", line 117, in acquire me = _get_ident() TypeError: 'NoneType' object is not callable ``` On similar issues like this one, it was understood that the interpreter was shutting down so builtins would start being set to `None`, but this seems like is not the case even though the error is very similar (see Issue #22) They appear when there is nothing else being executed remotely and the gateway is being closed.