From kelleyk at kelleyk.net Tue Oct 27 17:30:43 2015 From: kelleyk at kelleyk.net (Kevin Kelley) Date: Tue, 27 Oct 2015 14:30:43 -0700 Subject: [execnet-dev] Interest in pull requests? Message-ID: Howdy! While working on a little project of mine, I've implemented two new execnet-related features, and I'm curious if the maintainers would be receptive to pull requests for them or if you guys think they would be better kept separate. The first ("execnet-importhook") inserts itself into the import machinery on the remote/slave side. When an import is about to fail (because the module is not present on the slave), the slave asks the master if the master can find the module. If it can, the master sends the slave the module's source; the slave can then successfully import it. Right now this feature works with both absolute and relative imports. It only works with pure-Python modules (not with extension modules), though adding support for those (in situations where the master and the slave are similar enough to be using the same extension modules) would probably not be hard. It currently supports Python 3.4 and Python 3.5 on the slave side. Adding support for Python 2 is most likely possible; I would just have to spend some time remembering exactly how the import machinery works there, since it is rather different. (I'll probably wind up doing this eventually.) The current implementation doesn't change the source of execnet itself at all; there's just an extra call right after you create the gateway. The second ("execnet-sudo") uses pexpect to watch for (and respond to) password prompts so that you can use sudo on the remote/slave side to launch a privileged interpreter. This requires that PopenPiped() also pipe the process's stderr and that a hook be inserted into Popen2IOMaster.__init__() between the call to PopenPiped() and Popen2IO.__init__(). The hook calls a function that, generically, handles any connection setup; in this case, it's about 10 lines of pexpect code. I haven't tested exhaustively, but the current implementation seems to work well with a good mix of Python 2 and Python 3 interpreters and Ubuntu and OS X hosts. Let me know if you think it's worth my putting together pull requests for any of this! Cheers, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Thu Oct 29 05:00:00 2015 From: holger at merlinux.eu (holger krekel) Date: Thu, 29 Oct 2015 09:00:00 +0000 Subject: [execnet-dev] Interest in pull requests? In-Reply-To: References: Message-ID: <20151029090000.GD28212@merlinux.eu> Hi Kevin, On Tue, Oct 27, 2015 at 14:30 -0700, Kevin Kelley wrote: > Howdy! > > While working on a little project of mine, I've implemented two new > execnet-related features, and I'm curious if the maintainers would be > receptive to pull requests for them or if you guys think they would be > better kept separate. > > The first ("execnet-importhook") inserts itself into the import machinery > on the remote/slave side. When an import is about to fail (because the > module is not present on the slave), the slave asks the master if the > master can find the module. If it can, the master sends the slave the > module's source; the slave can then successfully import it. > > Right now this feature works with both absolute and relative imports. It > only works with pure-Python modules (not with extension modules), though > adding support for those (in situations where the master and the slave are > similar enough to be using the same extension modules) would probably not > be hard. It currently supports Python 3.4 and Python 3.5 on the slave > side. Adding support for Python 2 is most likely possible; I would just > have to spend some time remembering exactly how the import machinery works > there, since it is rather different. (I'll probably wind up doing this > eventually.) > > The current implementation doesn't change the source of execnet itself at > all; there's just an extra call right after you create the gateway. I wonder if we could introduce pluggy (a little one-module plugin system used by pytest and some other projects) and your importhook would become a plugin. This allows you to update/fix things and also initially only support 3.4/3.5 if you like. Pluggy basically allows a program to add 1:N function calls (N = number of plugins implementing a particular function). > The second ("execnet-sudo") uses pexpect to watch for (and respond to) > password prompts so that you can use sudo on the remote/slave side to > launch a privileged interpreter. > > This requires that PopenPiped() also pipe the process's stderr and that a > hook be inserted into Popen2IOMaster.__init__() between the call to > PopenPiped() and Popen2IO.__init__(). The hook calls a function that, > generically, handles any connection setup; in this case, it's about 10 > lines of pexpect code. > > I haven't tested exhaustively, but the current implementation seems to work > well with a good mix of Python 2 and Python 3 interpreters and Ubuntu and > OS X hosts. > > Let me know if you think it's worth my putting together pull requests for > any of this! Can you do PRs to show ronny and me (the current maintainers of execnet) the way how you implemented the two functionalities? This is just to allow us to see what changed etc, not primarily aimed at a direct merge. cheers, holger > Cheers, > > Kevin > _______________________________________________ > execnet-dev mailing list > execnet-dev at python.org > https://mail.python.org/mailman/listinfo/execnet-dev -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu