[py-svn] r66117 - in py/extradoc/talk/ep2009: execnet pytest rapid-testing rapid-testing/ui
hpk at codespeak.net
hpk at codespeak.net
Sat Jul 4 15:44:07 CEST 2009
Author: hpk
Date: Sat Jul 4 15:44:04 2009
New Revision: 66117
Added:
py/extradoc/talk/ep2009/pytest/
Removed:
py/extradoc/talk/ep2009/rapid-testing/ui/
Modified:
py/extradoc/talk/ep2009/execnet/execnet.txt
py/extradoc/talk/ep2009/execnet/model1.graffle
py/extradoc/talk/ep2009/execnet/model1.png
py/extradoc/talk/ep2009/rapid-testing/rapid-testing.txt
py/extradoc/talk/ep2009/rapid-testing/test_cached_setup.py
py/extradoc/talk/ep2009/rapid-testing/test_generate_tests.py
Log:
snapshot talk and tutorial material
Modified: py/extradoc/talk/ep2009/execnet/execnet.txt
==============================================================================
--- py/extradoc/talk/ep2009/execnet/execnet.txt (original)
+++ py/extradoc/talk/ep2009/execnet/execnet.txt Sat Jul 4 15:44:04 2009
@@ -1,29 +1,19 @@
-.. .. include:: beamerdefs.txt
-.. .. include:: <s5defs.txt>
+.. include:: beamerdefs.txt
+.. include:: <s5defs.txt>
=================================================================
Distributed Programming with py.execnet
=================================================================
-my technical background
-===========================
-
-- programming since 20 years
-- Python since around 2000
-- released projects: pypy, py.test/py lib, rlcompleter2
-- other: mailwitness, shpy, vadm, codespeak, ...
-- merlinux GmbH since 2004
-- PyPy EU-project 2004-2007
-
my distributed systems background
=============================================
+- programming since 20 years, Python since around 2000
- Adaptive Communication Environment (ACE)
-- Corba ORB TAO
-- Corba Transaction Service (XOTS)
-- automated system maintenance (service hosting)
+- Corba ORB TAO, C++ Corba Transaction Service (XOTS)
+- automated remote system maintenance
- py.test distribution of tests
-- consultancies on distributed systems
+- consultancies on RMI / distributed (C++/Java) systems
This talk
@@ -31,63 +21,110 @@
* intro to py.execnet, examples, demos
-* comparison with multiprocessing and RMI
+* some comparison notes (multiprocessing / RMI systems)
-* discussion about challenges / promises of
- execnet-style deployment models
+* summary / future bits
-Intro to py.execnet
-============================
-
-execnet does:
-
-1. instantiating remote Python processes
-2. sending code for remote execution
-3. send/receive of data between remote and local code
-4. (closing down remote process)
-
The basic model / terminology
===============================
.. image:: model1.png
- :scale: 100
+ :scale: 80
:align: center
-Interactive Example
+
+What **execnet** does
============================
+1. instantiate remote Python process
+2. send code for remote execution
+3. send/receive of data between remote and local code
+
+
+elastic code: gateway's remote_exec
+======================================
+
::
>>> gw = py.execnet.PopenGateway()
>>> src = "import os ; channel.send(os.getpid())"
>>> channel = gw.remote_exec(src)
>>> remote_pid = channel.getpid()
+
+What about remote failures?
+============================
+``channel.RemoteError`` textually
+represents remote traceback
-gateway's remote_exec
-==================================
+
+Channel send/receive
+========================================
-::
+* channels have symmetric send/receive
+* data items are Python primitive types (dict, int, string, ...)
- def remote_exec(source):
- """return channel object for communicating with the
- asynchronously executing 'source' code which will
- have a corresponding 'channel' object in its
- executing namespace."""
-Channel objects, send() and receive()
+Gateways
========================================
-symmetric send/receive, example::
+* gateways are represented on both sides
+* each gateway object spawns a receiver thread
+* but execution is single-threaded by default
+
+
+
+
+Gateway / process instantiation
+=======================================
+
+currently three gateway types:
- >>> src = "channel.send(channel.receive() + 1 )"
- >>> ch = gw.remote_exec(src)
- >>> ch.send(41)
- >>> result = ch.receive()
+* PopenGateway opens a local Subprocess
+
+* SshGateway opens a ssh-mediated remote subprocess
+
+* SocketGateway opens a socket-mediated remote subprocess
+
+Gateways work **across Python versions**, for example:
+a Python-2.4 process can connect with a Python-2.6 subprocess
+
+
+SshGateway zero-install deployment
+=======================================
+
+- uses underlying "ssh" command line tool
+- only requires plain Python Interpreter
+- no prior installation of remote code
+- sends code *elastically*
+
+example: getting remote machine info
+========================================
+
+- single-file script to gather info from remote host
+
+example: svn-hotsyncing repositories
+========================================
+
+- single-file script to hot-sync svn repository
+
+"elastic code"
+===============================
+
+.. image:: model2.png
+ :scale: 60
+ :align: center
+
+Some helpful "on-top" API
+=============================================
+
+.. image:: img/end_of_a_age_by_marikaz.jpg
+ :scale: 100
+ :align: left
+
+http://marikaz.deviantart.com/ CC 3.0 AN-ND
-**channels work with builtin basic python data structures**:
-dictionaries, lists, tuples, numbers, floats, ...
Channel objects, waitclose()
=====================================
@@ -100,10 +137,11 @@
f.write(content)
f.close()
""") # channel.close() is called automatically
- >>> ch.send(("hello.txt", "world")
+ >>> ch.send("hello.txt", "world")
>>> ch.waitclose()
-sync-receive data from sub processes
+
+receive data from multiple sub processes
=============================================
Use MultiChannels for receiving multiple results from remote code::
@@ -132,6 +170,7 @@
>>> res1 + res2
3
+
remote_exec is single threaded by default
========================================================
@@ -140,7 +179,7 @@
ch1 = gw.remote_exec("channel.send(channel.receive()+1)")
ch2 = gw.remote_exec("channel.send(channel.receive()+1)")
ch2.send(1)
- ch2.receive() # XXX BLOCKS of single remote thread !
+ ch2.receive() # XXX BLOCKS XXX
concurrent remote execution
@@ -159,57 +198,10 @@
[quick look into remote_init_threads implementation]
-Gateway / process instantiation
-=======================================
-
-currently three gateway types:
-
-* PopenGateway opens a local Subprocess
-
-* SshGateway opens a ssh-mediated remote subprocess
-
-* SocketGateway opens a socket-mediated remote subprocess
-
-Gateways can cross Python versions, for example:
-a Python-2.4 process can connect with a Python-2.6 subprocess
-
-
-SshGateway
-=======================================
-
-- uses underlying "ssh" command line tool
-
-- configurable: identity, ssh_config file,
- remote python interpreter
-
-- **zero-install ad-hoc deployment**:
-
- 1. instantiate (remote) python interpreter process
- 2. send dispatcher bootstrap code
- 3. dispatcher waits for remote_exec commands,
- manages channel communication
-
-SocketGateway
-=======================================
-
-- uses standard socket connections
-- needs a simple socket listen/accept/read loop on the remote side
-
-- **zero-install deployment**:
-
-1. **establish connection to existing remote-python interpreter process**
-2. send it bootstrap code that installs a gateway dispatcher
-3. dispatcher executes remote_exec commands, manages channel communication
-
-
XSpecs and py.execnet.makegateway()
===============================================
-1.0 introduces a string-scheme for specifying gateways::
-
- key1[=value1]//key2[=value2]//...
-
-Examples::
+1.0 introduces a "XSpec" string-scheme for specifying gateways::
popen//python=2.5//nice=20
socket=192.168.1.4:8888
@@ -219,106 +211,80 @@
>>> gateway = py.execnet.makegateway(xspec)
-e.g. used by ``py.test --tx ssh=remote_host``.
-
-zero-install / rsyncing dependencies
+rsyncing dependencies
=======================================
-**zero-install** means: no prior remote installation of code
-
-use ``py.execnet.RSync`` to transfer dependencies::
+use ``py.execnet.RSync`` to transfer packages::
>>> rsyncer = py.execnet.RSync("sourcedir")
>>> gw = py.execnet.SshGateway("codespeak.net")
>>> rsyncer.addtarget(gw)
+ >>> # add multiple targets
>>> rsyncer.send()
-detecting if we are run through remote_exec
-=============================================
+example: py.test distributed testing
+==========================================
::
- if __name__ == '__channelexec__':
- # we will also have a 'channel' object
-
-
-real life: system admin examples
-===================================
-
-- getting resource information from multiple remote linux machines
-
-- hot-syncing an SVN repository
-
-- py.test test-distribution
-
-- (other scripting)
-
-(demo + source code looking)
-
-
-Execnet Summary
-===============================
+ py.test --dist=each \
+ --tx ssh=wyvern \
+ --tx ssh=code \
+ --rsync mypkg
-* ad-hoc instantiate remote Python processes, represent as ``Gateway``
-
-* send and execute code remotely via ``gateway.remote_exec()``
-
-* communication through symmetric ``channel.receive()/channel.send()``
-
--> easy deployment and communication model
-
-multiprocessing
+multiprocessing (Python 2.6)
============================================
-- part of standard lib since CPython 2.6
-- tries to make thread/process distintinction transparent
-- needs **pickling** to work for resources to send/receive
-- recommends to use Pipes (one-to-one) and Queues (many-to-many) for communication
-- (also recommends to keep shared state minimal)
-
-XXX
-multiprocessing and ssh
-==============================
-
+- aims for transparent thread/process distintinction
+- requires **pickling** to work for your app
+- Pipes for one-to-one and Queues for many-to-many commnunication
- requires compatible "multiprocessing" package on the "remote" side
-- requires to-be-executed code to pre-exist/be importable on other side
-- multiprocessing.distributing helps with dispatching/managing cluster tasks
+- requires to-be-executed code to **pre-exist**
- Python version agnostic? there are backported multiprocessing packages
-XXX
+Remote Method Invocation
+==============================
+- use object identity models
+- static code distribution and static server code
+- usually works with pickling
+- pre-existing code on both sides
-Pyro / RMI
-===========
-
+the "pre-existing" code requirement ...
+=========================================
+- version-dependencies
+- manual installation
+- difficult to extend to "arbitrary" hosts
+- difficult to automate tests
+- **not developer friendly**
-Data protocol
-=====================
-chat/text based protocols
+Execnet Summary
+===============================
-examples: SMTP, HTTP, DNS, ...
+* ad-hoc instantiate remote Python processes
-**allows many implementation (languages)**
+* send and execute code **elastically**
-Data format
-===================
+* zero-install - code is determined client-side
-open or reverse engeneered specifications
+* communicate through symmetric channels
-pdf, xml, html, ODS, doc, ...
+* works across host and Python interpreter barriers
-**allows many implementation (languages)**
-**agnostic to transport mechanism**
+Open issues
+=============
-Remote method invocation
-==============================
+* improved debugging / tested teardown mechanisms
+* channels/gateways to cross multiple hosts
+* best practise model for sharing data
+* **user contributed VMs for elastic execution**
+holger @ merlinux.eu
-
Modified: py/extradoc/talk/ep2009/execnet/model1.graffle
==============================================================================
--- py/extradoc/talk/ep2009/execnet/model1.graffle (original)
+++ py/extradoc/talk/ep2009/execnet/model1.graffle Sat Jul 4 15:44:04 2009
@@ -35,21 +35,75 @@
<key>GraphicsList</key>
<array>
<dict>
+ <key>Bounds</key>
+ <string>{{337.674, 92.716}, {119, 17}}</string>
+ <key>Class</key>
+ <string>ShapedGraphic</string>
+ <key>FitText</key>
+ <string>YES</string>
+ <key>FontInfo</key>
+ <dict>
+ <key>Font</key>
+ <string>Helvetica</string>
+ <key>Size</key>
+ <real>12</real>
+ </dict>
+ <key>ID</key>
+ <integer>57</integer>
+ <key>Line</key>
+ <dict>
+ <key>ID</key>
+ <integer>55</integer>
+ <key>Position</key>
+ <real>0.72225594520568848</real>
+ <key>RotationType</key>
+ <integer>0</integer>
+ </dict>
+ <key>Shape</key>
+ <string>Rectangle</string>
+ <key>Style</key>
+ <dict>
+ <key>fill</key>
+ <dict>
+ <key>Draws</key>
+ <string>NO</string>
+ </dict>
+ <key>shadow</key>
+ <dict>
+ <key>Draws</key>
+ <string>NO</string>
+ </dict>
+ <key>stroke</key>
+ <dict>
+ <key>Draws</key>
+ <string>NO</string>
+ </dict>
+ </dict>
+ <key>Text</key>
+ <dict>
+ <key>Text</key>
+ <string>{\rtf1\mac\ansicpg10000\cocoartf824\cocoasubrtf480
+{\fonttbl\f0\fswiss\fcharset77 Helvetica;}
+{\colortbl;\red255\green255\blue255;}
+\pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\qc\pardirnatural
+
+\f0\fs28 \cf0 Python Execution}</string>
+ </dict>
+ </dict>
+ <dict>
<key>Class</key>
<string>LineGraphic</string>
<key>Head</key>
<dict>
<key>ID</key>
- <integer>46</integer>
- <key>Info</key>
- <integer>2</integer>
+ <integer>42</integer>
</dict>
<key>ID</key>
<integer>55</integer>
<key>Points</key>
<array>
- <string>{302.863, 112.386}</string>
- <string>{394.37, 162.425}</string>
+ <string>{399.331, 70.8661}</string>
+ <string>{396.345, 112.887}</string>
</array>
<key>Style</key>
<dict>
@@ -66,7 +120,7 @@
<key>Tail</key>
<dict>
<key>ID</key>
- <integer>51</integer>
+ <integer>41</integer>
</dict>
</dict>
<dict>
@@ -83,8 +137,8 @@
<integer>54</integer>
<key>Points</key>
<array>
- <string>{255.688, 112.386}</string>
- <string>{158.386, 162.425}</string>
+ <string>{153.425, 113.886}</string>
+ <string>{153.425, 170.079}</string>
</array>
<key>Style</key>
<dict>
@@ -101,7 +155,7 @@
<key>Tail</key>
<dict>
<key>ID</key>
- <integer>51</integer>
+ <integer>44</integer>
</dict>
</dict>
<dict>
@@ -118,9 +172,9 @@
<integer>49</integer>
<key>Points</key>
<array>
- <string>{189.213, 180.425}</string>
+ <string>{184.252, 188.079}</string>
<string>{283.465, 170.079}</string>
- <string>{363.543, 180.425}</string>
+ <string>{354.331, 202.252}</string>
</array>
<key>Style</key>
<dict>
@@ -144,7 +198,7 @@
</dict>
<dict>
<key>Bounds</key>
- <string>{{363.543, 162.425}, {61.6536, 36}}</string>
+ <string>{{354.331, 184.252}, {61.6536, 36}}</string>
<key>Class</key>
<string>ShapedGraphic</string>
<key>ID</key>
@@ -171,7 +225,7 @@
</dict>
<dict>
<key>Bounds</key>
- <string>{{127.559, 162.425}, {61.6535, 36}}</string>
+ <string>{{122.598, 170.079}, {61.6535, 36}}</string>
<key>Class</key>
<string>ShapedGraphic</string>
<key>ID</key>
@@ -211,7 +265,7 @@
<key>Points</key>
<array>
<string>{198.425, 95.3858}</string>
- <string>{363.543, 95.3858}</string>
+ <string>{354.331, 52.8661}</string>
</array>
<key>Style</key>
<dict>
@@ -237,7 +291,7 @@
</dict>
<dict>
<key>Bounds</key>
- <string>{{363.543, 77.3858}, {90, 36}}</string>
+ <string>{{354.331, 34.8661}, {90, 36}}</string>
<key>Class</key>
<string>ShapedGraphic</string>
<key>ID</key>
@@ -264,7 +318,7 @@
</dict>
<dict>
<key>Bounds</key>
- <string>{{354.331, 56.6929}, {99.2126, 184.252}}</string>
+ <string>{{340.158, 113.386}, {99.2126, 184.252}}</string>
<key>Class</key>
<string>ShapedGraphic</string>
<key>ID</key>
@@ -319,7 +373,7 @@
</dict>
<dict>
<key>Bounds</key>
- <string>{{256.237, 136.108}, {61, 68}}</string>
+ <string>{{251.754, 135.988}, {61, 68}}</string>
<key>Class</key>
<string>ShapedGraphic</string>
<key>FitText</key>
@@ -378,7 +432,7 @@
</dict>
<dict>
<key>Bounds</key>
- <string>{{226, 78.3858}, {108, 34}}</string>
+ <string>{{217.781, 56.6069}, {121, 34}}</string>
<key>Class</key>
<string>ShapedGraphic</string>
<key>FitText</key>
@@ -402,7 +456,7 @@
<key>ID</key>
<integer>39</integer>
<key>Position</key>
- <real>0.49403908848762512</real>
+ <real>0.51220786571502686</real>
<key>RotationType</key>
<integer>0</integer>
</dict>
@@ -429,7 +483,7 @@
{\colortbl;\red255\green255\blue255;}
\pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\qc\pardirnatural
-\f0\fs28 \cf0 remote-execute\
+\f0\fs28 \cf0 send and execute\
code}</string>
</dict>
</dict>
@@ -551,7 +605,7 @@
</dict>
</array>
<key>ModificationDate</key>
- <string>2009-06-26 12:50:42 +0200</string>
+ <string>2009-07-02 07:15:11 +0200</string>
<key>Modifier</key>
<string>holger krekel</string>
<key>NotesVisible</key>
@@ -626,9 +680,9 @@
<key>DrawerWidth</key>
<real>209</real>
<key>Frame</key>
- <string>{{1499, 82}, {574, 742}}</string>
+ <string>{{412, 168}, {574, 578}}</string>
<key>VisibleRegion</key>
- <string>{{0, 0}, {559, 628}}</string>
+ <string>{{0, 0}, {559, 464}}</string>
<key>Zoom</key>
<real>1</real>
</dict>
Modified: py/extradoc/talk/ep2009/execnet/model1.png
==============================================================================
Binary files. No diff available.
Modified: py/extradoc/talk/ep2009/rapid-testing/rapid-testing.txt
==============================================================================
--- py/extradoc/talk/ep2009/rapid-testing/rapid-testing.txt (original)
+++ py/extradoc/talk/ep2009/rapid-testing/rapid-testing.txt Sat Jul 4 15:44:04 2009
@@ -5,6 +5,16 @@
Rapid testing with py.test
=================================================================
+these slides
+===========================
+
+http://codespeak.net/~hpk/rapid-testing.pdf
+
+http://tinyurl.com/rapid-testing
+
+install: ``easy_install -U py``
+
+latest version: 1.0.0b6 today :-)
my testing background
===========================
Modified: py/extradoc/talk/ep2009/rapid-testing/test_cached_setup.py
==============================================================================
--- py/extradoc/talk/ep2009/rapid-testing/test_cached_setup.py (original)
+++ py/extradoc/talk/ep2009/rapid-testing/test_cached_setup.py Sat Jul 4 15:44:04 2009
@@ -10,7 +10,7 @@
return request.cached_setup(
setup=Myarg,
teardown=lambda arg: arg.finalize(),
- scope="function",
+ scope="module",
)
def test_hello(myarg):
Modified: py/extradoc/talk/ep2009/rapid-testing/test_generate_tests.py
==============================================================================
--- py/extradoc/talk/ep2009/rapid-testing/test_generate_tests.py (original)
+++ py/extradoc/talk/ep2009/rapid-testing/test_generate_tests.py Sat Jul 4 15:44:04 2009
@@ -1,12 +1,11 @@
-def pytest_generate_tests(metafunc):
- if "url" in metafunc.funcargnames:
- for val in ('testrun.org', 'codespeak.net'):
- metafunc.addcall(id=val, funcargs={'url': 'http://' + val})
+def test_function(param):
+ assert param < 3
+def pytest_generate_tests(metafunc):
+ if "param" not in metafunc.funcargnames:
+ return
+ for i in range(4):
+ funcargs = {'param': i}
+ metafunc.addcall(funcargs=funcargs)
-import urllib
-def test_open(url):
- f = urllib.urlopen(url)
- assert f
- f.close()
More information about the pytest-commit
mailing list