[py-svn] r65934 - py/extradoc/talk/ep2009/pytest-advanced
hpk at codespeak.net
hpk at codespeak.net
Wed Jun 24 21:22:15 CEST 2009
Author: hpk
Date: Wed Jun 24 21:22:12 2009
New Revision: 65934
Modified:
py/extradoc/talk/ep2009/pytest-advanced/pytest-advanced.txt
Log:
mostly finalize the full tutorial draft
Modified: py/extradoc/talk/ep2009/pytest-advanced/pytest-advanced.txt
==============================================================================
--- py/extradoc/talk/ep2009/pytest-advanced/pytest-advanced.txt (original)
+++ py/extradoc/talk/ep2009/pytest-advanced/pytest-advanced.txt Wed Jun 24 21:22:12 2009
@@ -240,19 +240,23 @@
get-going exercise (max 10 min)
============================================
-install with one of:
+- install with one of:
+
+ - ``easy_install -U py``
+ - svn checkout http://codespeak.net/svn/py/trunk
+ - hg clone https://bitbucket.org/hpk42/py-trunk/
+ - run "python setup.py" with "install" or "develop"
+
+- create a ``test_file.py`` with some ``test_function``
+
+- execute ``py.test test_file.py``
-- easy_install "py"
-- svn checkout http://codespeak.net/svn/py/trunk
-- hg clone https://bitbucket.org/hpk42/py-trunk/
-- run "python setup.py" with "install" or "develop"
-- create a ``test_file.py`` and run ``py.test test_file.py``
- help your neighbours!
-selected options
+some important basic options
==============================
-among others, ``py.test --help`` yields these useful options:
+among others, ``py.test --help`` shows these options:
- -s - disable catching of stdout/stderr during test run.
- -x/--exitfirst - exit instantly on first error or failed test.
@@ -276,19 +280,18 @@
observations and the setup question
==========================================
-* test values and configuration mixes with test code
+* test configuration mixes with test code
* importing 'app' may fail
* the ``app.pkg.SomeClass`` reference may change
-what if multiple test functions have similar setup,
-maybe with slightly different configuration for
-class instantiation?
+what if multiple test functions have class setup?
"unittest.py" style setup / abstract view
==========================================
::
+
import unittest
from app.pkg import SomeClass
@@ -304,27 +307,28 @@
==========================================
* **multiple methods can reuse the same setup**
-* ``self.inst1`` instantiated anew for each test invocation
+* ``self.inst1`` instantiated a new for each test invocation
* test values / configuration mixes with test code
* the ``app.pkg.SomeClass`` reference may change
* **functions now group by setup/fixture code**
-old-style xUnit py.test extensions
+sidenote: old-style xUnit py.test extensions
==========================================================
-to help with managing test state across modules, classes,
-and methods, py.test invented:
+py.test was the first to introduce extensions:
- setup_module(module) / teardown_module(module)
+- setup_module / teardown_module
- setup_class(cls) / teardown_class(cls)
+- setup_class / teardown_class
- setup_method(self, method) / teardown_method(self, method)
+- setup_method / teardown_method
-but:
-- even increases need for grouping of functions by setup
-- test values / configuration still mix with test code
-- ... mostly deprecated!
+but it didn't change the issues:
+
+- test configuration still mixes with test code
+- cements grouping of functions per-file and setup
+
+**not recommended anymore**
meet "funcargs" - test function arguments
@@ -335,57 +339,34 @@
def test_something(inst1):
assert inst1.method() == "ok"
-or::
-
class TestGroup:
- def test_other(self, inst1):
- assert inst1.method() == "ok"
+ def test_method_equal(self, inst1, inst2):
+ assert inst1.method() == inst2.othermethod()
observations
==========================================
-* test values are simply used in test code
+* test code uses already setup ``inst1`` instance
* no imports or app.pkg.SomeClass references
* freedom to group tests logically
* multiple functions re-use same funcarg setup
-How to setup the funcarg value?
+How do funcarg values get setup?
==========================================
-write down in test module, ``conftest.py`` (or a plugin)::
-
- from app.pkg import SomeClass
- def pytest_funcarg__inst1(request):
- return SomeClass("somevalue")
+write down in test module, ``conftest.py`` or registered plugin::
-observations / notes
-===================================================
+ from app.pkg import SomeClass
-* test value setup separated from test code
+ def pytest_funcarg__inst1(request):
+ return SomeClass("somevalue")
-* funcarg setup function automatically discovered by looking
- for ``pytest_funcarg__`` prefixed functions
+* automatically discovered by naming convention
* app specific bootstraping contained in one place
* re-useable from test functions across a whole project
-let's introduce a command line option
-===================================================
-::
-
- # ./tests/conftest.py
- def pytest_addoption(parser):
- parser.addoption("--val", action="store")
- def pytest_funcarg_inst1(request):
- return SomeClass(request.config.getvalue("val"))
-
-observations
-===================================================
-
-* test configuration separated from test code
-* app bootstraps for testing in one place
-* can be used from any test function anywhere
``request`` object attributes
===================================================
@@ -401,54 +382,26 @@
Exercise (max 10 minutes)
==========================================
-* write a new package "mypkg"
+* create a new package directory "mypkg"
* add mypkg/__init__ and mypkg/test_url.py
* add a ``test_open(url)`` function checking
if URL can be opened
-* write the provider in mypkg/conftest.py
+* write ``pytest_funcarg__url`` provider in mypkg/conftest.py
* run your test
-* optional: add a command line option "--url"
-
-Bonus: add more tests, play with wrongly named args, introduce options.
-
-invoking a test function with different values
-================================================
-
-::
-
- def pytest_generate_tests(metafunc):
- if "url" in metafunc.funcargnames:
- metafunc.addcall(funcargs=dict(url="http://testrun.org"))
- metafunc.addcall(funcargs=dict(url="https://codespeak.net"))
-
-``metafunc`` attributes and abilities
-================================================
-
-``metafunc.funcargnames``: set of required function arguments for given function
-
-``function, cls, module, config``: same as request object
-
-means: you can easily implement a decorator or per-module
-data for specifiying argument sets. see
-
-deprecation sidenote on yield-generated tests
-================================================
+* optional: add a second funcarg, misspell funcarg names
-py.test pioneered 'yield' based testing
-deprecated since 1.0 because it's a very limited form of the
-new ``pytest_generate_tests()`` mechanism.
-
-
-Calling a finalizer for a funcarg
+registering a finalizer for a funcarg
=============================================
-a funcarg provider can use its ``request`` object to
-register a finalization call::
+::
- myfile = open(...)
- request.addfinalizer(lambda: myfile.close())
+ def pytest_funcarg__myfile(request):
+ myfile = open(...)
+ request.addfinalizer(lambda: myfile.close())
+ return myfile
+[interactive demo]
Managing more complex scope setup
=============================================
@@ -457,9 +410,10 @@
- some funcargs need explicit teardown
- managing this correctly can be complex
-py.test provides a powerful high-level helper::
+py.test provides a powerful high-level helper function::
- ``request.cached_setup(...)``
+ def pytest_funcarg__arg(request):
+ return request.cached_setup(setup, teardown, scope)
Example for caching a Database object Setup
@@ -474,21 +428,90 @@
scope="session",
)
-* the test code does not need to know about this optimization
-* easy to change to per-module or to per-function setup/teardown!
+* test code simply uses ``db`` function argument
+* test code does not know about setup/teardown details
+* test code does not know about caching scopes
+* easy to change caching scope
+* **ideal for functional testing**
* **interactive demo**
+the mysetup funcarg pattern
+============================================
+
+avoid the need to have one funcarg for each app object::
+
+ def pytest_funcarg__mysetup(request):
+ return MySetup(request)
+
+ class MySetup:
+ def __init__(self, request):
+ self.config = request.config
+
+ def obj1(self):
+ return ...
+ def obj2(self):
+ return ...
+
+invoking a test function with different values
+================================================
+
+::
+
+ def pytest_generate_tests(metafunc):
+ if "url" not in metafunc.funcargnames:
+ return
+ metafunc.addcall(funcargs={'url': "http://testrun.org"})
+ metafunc.addcall(funcargs={'url': "http://xyz.net"])
+
+
+[interactive demo]
+
+``metafunc`` attributes
+================================================
+
+``metafunc.funcargnames``: funcarg names for given test function
+
+``metafunc.function``: python test function
+
+``metafunc.cls``: containing test class
+
+``metafunc.module``: test module
+
+``metafunc.config``: access to options and general config
+
+pytest_generate_tests for custom parametrization
+==================================================
+
+see blog post:
+
+ http://tetamap.wordpress.com/2009/05/13/parametrizing-python-tests
+
+or in short:
+
+ http://tinyurl.com/ofry94
+
+
+deprecation sidenote on yield-generated tests
+================================================
+
+- py.test pioneered 'yield' test generators
+
+- deprecated since 1.0 because:
+ - limited value
+ - was hackish to implement
+ - ``pytest_generate_tests()`` is vastly superior
+
+
summary of py.test funcargs
================================
-* test function is fully separated from test setup/bootstrap
-* test function can focus on specific testing aspects
-* test functions can be called multiple times
+* test function separated from configuration and setup
+* **cleaner, more focused test functions**
+* test functions can be called with multiple parameter sets
* manage simple and complex "funcarg" values
* re-use test functions with generated calls
-* ideal for functional testing
-
+* ideal for functional or system testing
Break
=====
@@ -500,7 +523,7 @@
http://marikaz.deviantart.com/ CC 3.0 AN-ND
-Using Plugins and Extensions
+Hooks and Extensions
=========================================
.. image:: img/end_of_a_age_by_marikaz.jpg
@@ -509,101 +532,102 @@
http://marikaz.deviantart.com/ CC 3.0 AN-ND
-Historic view
-==========================
+py.test 1.0 hooks
+===========================
+
+py.test calls more than 30 hooks to implement
-- 0.9.x uses conftest's for extension and configuration
-- 1.0 uses "plugins" for extending and conftest.py for configuration
-- we do "smooth" transition because of existing test code base
+- options and configuration
+- test collection
+- running of tests
+- reporting
+- (distribution of tests)
-Customizing py.test
+what are hooks?
===========================
-- configuration values go to conftest.py files
-- write local or global plugins
-- provide funcargs
-
-Conftest.py
-===============
-
-- can be put into test directory or higher up
-- contains test configuration values
-- specifies plugins to use
-- can provide default values for command line options
-
-Specifying plugins
-========================
-
-- ``-p NAME``: load comma-separated list of plugins
-- plugins are always named "pytest_NAME" and can be
- anywhere in your import path
+* hooks are python functions
+* hook functions alway start with ``pytest_``
+* strict hook signature checking when hooks are loaded
+* hooks are usually attributes of a **plugin** object
+* ``pytest_funcarg_*`` providers are hooks as well
-Writing a local conftest plugin
-====================================
+what are plugins?
+===========================
-you can write a local conftest.py based plugin::
+- ``conftest.py`` modules
+- ``pytest_*`` modules or packages.
- class ConftestPlugin:
- def pytest_addoption(self, parser):
- parser.addoption("--myworld", action="store_true")
- ...
+conftest.py modules
+=======================================
-Exercise
-==========================================
+- are automatically discovered
+- can be put in a test directory or higher up
+- can contain hook functions
-* add an option for specifying url on the command line
-* look at "py.test -h"
+**conftest.py: project specific config and extensions**
+named pytest_* plugins
+==============================
-other Plugin Examples
-=========================================
+- normal python modules or packages, normally imported
+- always named "pytest_*" all lowercase
+- ``-p NAME`` to load plugin(s) from the command line
-- integrate collection/run of traditional unit-tests
-- run functions in their own tempdir
-- testing ReST documents
-- running Prolog tests (old: conftest based)
-- running Javascript tests (old: conftest based)
-- html reporting for nightly runs (old: conftest-based)
-if time permits ...
-=========================================
+example: implementing a new py.test.* helper
+=================================================
-- let's play with "pytest_unittest" plugin
+::
+ # ./conftest.py or pytest_myname.py
+ def pytest_namespace(config):
+ return {'hello': 'world'}
-Writing Plugins (30 minutes)
-============================================================================
+makes ``py.test.hello`` available with value ``'hello'``
-- test collection objects
-- plugin hooks and events
-- event system for custom reporting
-- test collection hooks
-- test running hooks
-py.test collection objects
-=============================
+Plugin Examples
+=========================================
-* collectors, ``collect()`` returns list of "colitems"
-* items, implement ``runtest()`` for running a test
+- pytest_unittest.py: mark tests as "expected to fail"
+- pytest_xfail.py: mark test as "expected to fail" with py.test.xfail
-test collection tree
-========================
+- pytest_pocoo.py: send failure tracebacks to pocoo paste service
-* collection tree is built iteratively
-* test collection starts from directories or files (via cmdline)
+- pytest_monkeypatch.py: safely patch objects/environment
+
+- pytest_figleaf.py: generate html coverage reports
+- pytest_resultlog.py: generate buildbot-friendly output
-py.test.collect filesystem objects
-======================================
+Basic working of py.test
+==============================
+
+- collect tests
+- (distribute tests to remote processes)
+- run test
+- generate test report
+- (send basic report info)
+- report
-* Directory
-* File
+py.test collection objects
+=============================
+* collection tree is built iteratively
+* test collection starts from directories or files (via cmdline)
+* ``collector.collect()`` returns list of collected children
+* ``item.runtest()`` runs a test
+easily allows for non-python tests, examples:
-always available Attributes
+- pytest_restdoc.py for ReST syntax and remote URL checking
+
+- pytest_doctest.py for running doctests
+
+collection node Attributes
=======================================
**parent**: a reference to the collector that produced us
@@ -612,31 +636,14 @@
**config**: test configuration
-Python object collectors
-=======================================
-
-**obj** points to the underlying python object.
-- **Module**
-- **Class**/**Instance**
-- **Generator**
-- **Function**
-
-Exercise "collection tree"
+Quick Exercise "collection tree"
====================================
-inspecting the test collection tree::
+inspect your test collection tree::
py.test --collectonly
-Hooks and Events
-===================
-
-- Plugin hook methods implement interaction
- with configuration/collection/running of tests
-
-- Events are called for notification purposes,
- are not asked for return values.
Configuration Hooks
======================
@@ -651,8 +658,8 @@
def pytest_unconfigure(self, config):
""" called before test process is exited. """
-Collection Hooks
-======================
+Filesystem Collection Hooks
+===================================
::
def pytest_collect_file(self, path, parent):
@@ -661,62 +668,42 @@
def pytest_collect_directory(self, path, parent):
""" return Collection node or None. """
- def pytest_collect_recurse(self, path, parent):
- """ return True/False to cause/prevent recursion. """
-
Python collection hooks
=====================================
::
- def pytest_pymodule_makeitem(self, modcol, name, obj):
- """ return custom item/collector for a python object in a module, or None. """
+
+ def pytest_pycollect_makeitem(collector, name, obj):
+ """ return custom item/collector for a python object in a module, or None. """
function test run hooks
==========================
::
- def pytest_pyfunc_call(self, pyfuncitem, args, kwargs):
- """ return True if we consumed/did the call to the python function item. """
- def pytest_item_makereport(self, item, excinfo, when, outerr):
- """ return ItemTestReport event for the given test outcome. """
+ def pytest_runtest_setup(item):
+ def pytest_runtest_call(item):
+ def pytest_runtest_teardown(item):
-Reporting hooks
+reporting hooks
==========================
-::
- def pytest_report_teststatus(self, event):
- """ return shortletter and verbose word. """
-
- def pytest_terminal_summary(self, terminalreporter):
- """ add additional section in terminal summary reporting. """
-Event methods
-===============
-
-are only called, no interaction.
-
-see ``py/test/plugin/pytest_terminal.py``
-for a full selection of events.
-
-Warning
-===============
+::
-naming changes might take place before 1.0 final
+ report = hook.pytest_runtest_makereport(item, call)
+ pytest_runtest_logreport(report)
-Writing cross-project plugins
-==================================
-- put plugin into pytest_name.py or package
-- make a "NamePlugin" class available
-- release or copy to somewhere importable
+Terminal reporting hooks
+==========================
+::
-Exercise
-==================================
+ def pytest_report_teststatus(self, event):
+ """ return shortletter and verbose word. """
-* port conftest plugin to global "pytest_myapp" plugin.
-* put pytest_myapp somewhere where it's importable (or release it :)
-* put "pytest_plugins = 'pytest_myapp'" into your test module
+ def pytest_terminal_summary(self, terminalreporter):
+ """ add additional section in terminal summary reporting. """
-Distributed Testing (45 minutes)
+Distributed Testing (30 minutes)
====================================
.. image:: img/rails_in_the_city_by_marikaz.jpg
@@ -785,17 +772,16 @@
Assuming an IP address, you can now tell py.test to distribute: :
- py.test -d --tx socket=192.168.1.102:8888 --rsyncdir mypkg mypkg
+ py.test -d --tx socket=IPADDRESS:8888 --rsyncdir mypkg mypkg
Exercise: Move settings into conftest
========================================
-mypkg/conftest.py::
+conftest.py::
+
rsyncdirs = ['.']
pytest_option_tx = ['ssh=myhost', 'socket=...']
-mypkg/conftest.py::
- rsyncdirs = ['.']
Distribution modes
========================================
@@ -819,6 +805,9 @@
* asynchronously send and receive data between processes through channels
* completely avoid manual installation steps on remote places
+**see my thursday morning talk during EuroPython2009 :)**
+
+
xspecs: Exec environment specs
=================================
@@ -840,6 +829,7 @@
=================================
::
+
ssh=wyvern//python=python2.4//chdir=mycache
popen//python=2.5//nice=20
socket=192.168.1.4:8888
More information about the pytest-commit
mailing list