[py-svn] r63212 - py/trunk/py/doc
hpk at codespeak.net
hpk at codespeak.net
Mon Mar 23 00:18:53 CET 2009
Author: hpk
Date: Mon Mar 23 00:18:49 2009
New Revision: 63212
Modified:
py/trunk/py/doc/execnet.txt
Log:
completely revamp the execnet web page
Modified: py/trunk/py/doc/execnet.txt
==============================================================================
--- py/trunk/py/doc/execnet.txt (original)
+++ py/trunk/py/doc/execnet.txt Mon Mar 23 00:18:49 2009
@@ -1,57 +1,27 @@
-The py.execnet library
-======================
-
-.. contents::
-.. sectnum::
-
-A new view on distributed execution
------------------------------------
-
-``py.execnet`` supports ad-hoc distribution of parts of
-a program across process and network barriers. *Ad-hoc*
-means that the client side may completely control
-
-* which parts of a program execute remotely and
-
-* which data protocols are used between them
-
-without requiring any prior manual installation
-of user program code on the remote side. In fact,
-not even a prior installation of any server code
-is required, provided there is a way to get
-an input/output connection to a python interpreter
-(for example via "ssh" and a "python" executable).
-
-By comparison, traditional Remote Method Based (RMI)
-require prior installation and manual rather
-heavy processes of setup, distribution and
-communication between program parts.
-What about Security? Are you completely nuts?
----------------------------------------------
+``py.execnet`` allows to:
-We'll talk about that later :-)
+* instantiate local or remote Python Processes
+* send code for execution in one or many processes
+* asynchronously send and receive data between processes through channels
+* completely avoid manual installation steps on remote places
-Basic Features
-==============
-
-With ''py.execnet'' you get the means
-
-- to execute python code fragements in remote processes and
-- to interchange data between asynchronously executing code fragments
-
-
-Available Gateways
------------------------------------------
+.. contents::
-You may use one of the following connection methods:
+Gateways: immediately spawn local or remote process
+----------------------------------------------------
-* :api:`py.execnet.PopenGateway` a subprocess on the local
- machine. Useful for jailing certain parts of a program
- or for making use of multiple processors.
+In order to send code to a remote place or a subprocess
+you need to instantiate a so-called Gateway object.
+There are currently three Gateway classes:
+
+* :api:`py.execnet.PopenGateway` to open a subprocess
+ on the local machine. Useful for making use
+ of multiple processors to to contain code execution
+ in a separated environment.
-* :api:`py.execnet.SshGateway` a way to connect to
+* :api:`py.execnet.SshGateway` to connect to
a remote ssh server and distribute execution to it.
* :api:`py.execnet.SocketGateway` a way to connect to
@@ -59,13 +29,14 @@
requires a manually started
:source:py/execnet/script/socketserver.py
script. You can run this "server script" without
- having the py lib installed on that remote system.
+ having the py lib installed on the remote system
+ and you can setup it up as permanent service.
-executing code remotely
--------------------------------------
+remote_exec: execute source code remotely
+--------------------------------------------
-All gateways offer remote code execution via this high level function:
+All gateways offer remote code execution via this high level function::
def remote_exec(source):
"""return channel object for communicating with the asynchronously
@@ -76,7 +47,7 @@
side and get both a local and a remote Channel_ object,
which you can use to have the local and remote site
communicate data in a structured way. Here is
-an example:
+an example for reading the PID::
>>> import py
>>> gw = py.execnet.PopenGateway()
@@ -92,7 +63,7 @@
.. _`channel-api`:
.. _`exchange data`:
-Bidirectionally exchange data between hosts
+Channels: bidirectionally exchange data between hosts
-------------------------------------------------------------
A channel object allows to send and receive data between
@@ -131,44 +102,52 @@
A remote side blocking on receive() on this channel
will get woken up and see an EOFError exception.
-Instantiating a gateway from a string-specification
----------------------------------------------------------
-To specify Gateways with a String::
+.. _xspec:
- >>> import py
- >>> gw = py.execnet.makegateway("popen")
- >>> ex = gw.remote_exec("import sys ; channel.send(sys.executable)").receive()
- >>> assert ex == py.std.sys.executable, (ex, py.std.sys.executable)
- >>>
-current gateway types and specifications
-+++++++++++++++++++++++++++++++++++++++++++++++
+XSpec: string specification for gateway type and configuration
+-------------------------------------------------------------------
-+------------------------+-------------------------------------------+
-| ssh host | ssh:host:pythonexecutable:path |
-+------------------------+-------------------------------------------+
-| local subprocess | popen:python_executable:path |
-+------------------------+-------------------------------------------+
-| remote socket process | socket:host:port:python_executable:path |
-+------------------------+-------------------------------------------+
+``py.execnet`` supports a simple extensible format for
+specifying and configuring Gateways for remote execution.
+You can use a string speficiation to make a new gateway,
+for example a new SshGateway::
-examples of valid specifications
-++++++++++++++++++++++++++++++++++++++
+ gateway = py.execnet.makegateway("ssh=myhost")
-``ssh:wyvern:python2.4`` signifies a connection to a Python process on the machine reached via "ssh wyvern", current dir will be the 'pyexecnet-cache' subdirectory.
+Let's look at some examples for valid specifications.
+Specification for an ssh connection to `wyvern`, running on python2.4 in the (newly created) 'mycache' subdirectory::
-``socket:192.168.1.4`` signifies a connection to a Python Socket server process to the given IP on the default port 8888; current dir will be the 'pyexecnet-cache' directory.
+ ssh=wyvern//python=python2.4//chdir=mycache
-``popen:python2.5`` signifies a connection to a python2.5 subprocess; current dir will be the current dir of the instantiator.
+Specification of a python2.5 subprocess; with a low CPU priority ("nice" level). Current dir will be the current dir of the instantiator (that's true for all 'popen' specifications unless they specify 'chdir')::
-``popen::pytest-cache1`` signifies a connection to a subprocess using ``sys.executable``; current dir will be the `pytest-cache1`.
+ popen//python=2.5//nice=20
+Specification of a Python Socket server process that listens on 192.168.1.4:8888; current dir will be the 'pyexecnet-cache' sub directory which is used a default for all remote processes::
-Examples for execnet usage
--------------------------------------------
+ socket=192.168.1.4:8888
-Example: compare cwd() of Popen Gateways
+More generally, a specification string has this general format::
+
+ key1=value1//key2=value2//key3=value3
+
+If you omit a value, a boolean true value is assumed. Currently
+the following key/values are supported:
+
+* ``popen`` for a PopenGateway
+* ``ssh=host`` for a SshGateway
+* ``socket=address:port`` for a SocketGateway
+* ``python=executable`` for specifying Python executables
+* ``chdir=path`` change remote working dir to given relative or absolute path
+* ``nice=value`` decrease remote nice level if platforms supports it
+
+
+Examples fo py.execnet usage
+-------------------------------------
+
+compare cwd() of Popen Gateways
++++++++++++++++++++++++++++++++++++++++
A PopenGateway has the same working directory as the instantiatior::
@@ -180,10 +159,10 @@
>>> assert res == os.getcwd()
>>> gw.exit()
-Example: multichannels
-++++++++++++++++++++++++++++++++++++++++
+synchronously receive results from two sub processes
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-MultiChannels manage 1-n execution and communication:
+Use MultiChannels for receiving multiple results from remote code::
>>> import py
>>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
@@ -193,20 +172,33 @@
>>> assert len(l) == 2
>>> assert 1 in l
>>> assert 2 in l
-
-MultiGateways help with sending code to multiple remote places:
+
+assynchronously receive results from two sub processes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Use ``MultiChannel.make_receive_queue()`` for asynchronously receiving
+multiple results from remote code. This standard Queue provides
+``(channel, result)`` tuples which allows to determine where
+a result comes from::
>>> import py
- >>> mgw = py.execnet.MultiGateway([py.execnet.PopenGateway() for x in range(2)])
- >>> mch = mgw.remote_exec("import os; channel.send(os.getcwd())")
- >>> res = mch.receive_each()
- >>> assert res == [os.getcwd()] * 2, res
- >>> mgw.exit()
-
+ >>> ch1 = py.execnet.PopenGateway().remote_exec("channel.send(1)")
+ >>> ch2 = py.execnet.PopenGateway().remote_exec("channel.send(2)")
+ >>> mch = py.execnet.MultiChannel([ch1, ch2])
+ >>> queue = mch.make_receive_queue()
+ >>> chan1, res1 = queue.get() # you may also specify a timeout
+ >>> chan2, res2 = queue.get()
+ >>> res1 + res2
+ 3
+ >>> assert chan1 in (ch1, ch2)
+ >>> assert chan2 in (ch1, ch2)
+ >>> assert chan1 != chan2
-Example: receiving file contents from remote places
+receive file contents from remote SSH account
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-problem: retrieving contents of remote files::
+
+Here is a small program that you can use to retrieve
+contents of remote files::
import py
# open a gateway to a fresh child process
@@ -224,50 +216,20 @@
# process content
# later you can exit / close down the gateway
- contentgateway.exit()
+ gw.exit()
-A more complicated "nested" Gateway Example
-...........................................
+Instantiate a socket server in a new subprocess
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The following example opens a PopenGateway, i.e. a python
-child process, starts a socket server within that process and
-then opens a SocketGateway to the freshly started
-socketserver. Thus it forms a "triangle"::
-
-
- CLIENT < ... > PopenGateway()
- < .
- . .
- . .
- . .
- > SocketGateway()
-
-The below "socketserver" mentioned script is a small script that
-basically listens and accepts socket connections, receives one
-liners and executes them.
-
-Here are 20 lines of code making the above triangle happen::
-
+child process, and starts a socket server within that process
+and then opens a second gateway to the freshly started
+socketserver::
+
import py
- port = 7770
- socketserverbootstrap = py.code.Source(
- mypath.dirpath().dirpath('bin', 'socketserver.py').read(),
- """
- import socket
- sock = bind_and_listen(("localhost", %r))
- channel.send("ok")
- startserver(sock)
- """ % port)
- # open a gateway to a fresh child process
- proxygw = py.execnet.PopenGateway()
-
- # execute asynchronously the above socketserverbootstrap on the other
- channel = proxygw.remote_exec(socketserverbootstrap)
-
- # the other side should start the socket server now
- assert channel.receive() == "ok"
- gw = py.execnet.SocketGateway('localhost', cls.port)
- print "initialized socket gateway to port", cls.port
+ popengw = py.execnet.PopenGateway()
+ socketgw = py.execnet.SocketGateway.new_remote(popengw, ("127.0.0.1", 0))
+ print socketgw._rinfo() # print some info about the remote environment
More information about the pytest-commit
mailing list