[py-svn] r63200 - py/trunk/py/doc
hpk at codespeak.net
hpk at codespeak.net
Sun Mar 22 01:38:46 CET 2009
Author: hpk
Date: Sun Mar 22 01:38:43 2009
New Revision: 63200
Added:
py/trunk/py/doc/test-dist.txt
py/trunk/py/doc/test-examples.txt
py/trunk/py/doc/test-ext.txt
py/trunk/py/doc/test-features.txt
py/trunk/py/doc/test-quickstart.txt
Modified:
py/trunk/py/doc/contact.txt
py/trunk/py/doc/impl-test.txt
py/trunk/py/doc/test-config.txt
py/trunk/py/doc/test-plugins.txt
py/trunk/py/doc/test.txt
Log:
new docs about testing
refactoring of documentation
new entry page
Modified: py/trunk/py/doc/contact.txt
==============================================================================
--- py/trunk/py/doc/contact.txt (original)
+++ py/trunk/py/doc/contact.txt Sun Mar 22 01:38:43 2009
@@ -1,6 +1,8 @@
py lib contact and communication
===================================
+- You may also subscribe to `tetamap`_, where Holger Krekel
+ often posts about testing and py.test related news.
- **#pylib on irc.freenode.net**: you are welcome
to lurk or ask questions in this IRC channel, it also tracks py lib commits.
@@ -20,6 +22,8 @@
.. _`get an account`:
+.. _tetamap: http://tetamap.wordpress.com
+
get an account on codespeak
---------------------------
Modified: py/trunk/py/doc/impl-test.txt
==============================================================================
--- py/trunk/py/doc/impl-test.txt (original)
+++ py/trunk/py/doc/impl-test.txt Sun Mar 22 01:38:43 2009
@@ -131,28 +131,6 @@
py.test --traceconfig
-adding custom options
-+++++++++++++++++++++++
-
-To register a project-specific command line option
-you may have the following code within a ``conftest.py`` file::
-
- import py
- Option = py.test.config.Option
- option = py.test.config.addoptions("pypy options",
- Option('-V', '--view', action="store_true", dest="view", default=False,
- help="view translation tests' flow graphs with Pygame"),
- )
-
-and you can then access ``option.view`` like this::
-
- if option.view:
- print "view this!"
-
-The option will be available if you type ``py.test -h``
-Note that you may only register upper case short
-options. ``py.test`` reserves all lower
-case short options for its own cross-project usage.
customizing the collecting and running process
-----------------------------------------------
@@ -245,7 +223,7 @@
parameter sets but counting each of the calls as a separate
tests.
-.. _`generative tests`: test.html#generative-tests
+.. _`generative tests`: test-features.html#generative-tests
The other extension possibility is about
specifying a custom test ``Item`` class which
Modified: py/trunk/py/doc/test-config.txt
==============================================================================
--- py/trunk/py/doc/test-config.txt (original)
+++ py/trunk/py/doc/test-config.txt Sun Mar 22 01:38:43 2009
@@ -1,28 +1,23 @@
Test configuration
========================
-using / specifying plugins
--------------------------------
-
-you can instruct py.test to use additional plugins by:
-
-* setting the PYTEST_PLUGINS environment variable
- to a comma-separated list of plugins
-* XXX supplying "--plugins=NAME1,NAME2,..." at the command line
-* setting "pytest_plugins='name1', 'name2'" in
- ``conftest.py`` files or in python test modules.
+test options and values
+-----------------------------
-py.test will load all plugins along with their dependencies
-(plugins may specify "pytest_plugins" as well).
+You can see all available command line options by running::
-test option values
------------------------------
+ py.test -h
-py.test will lookup the value of an option "NAME" in this order:
+py.test will lookup values of options in this order:
* option value supplied at command line
* content of environment variable ``PYTEST_OPTION_NAME=...``
* ``name = ...`` setting in the nearest ``conftest.py`` file.
-This means that you can specify default options per-run,
-per shell session or per project directory.
+The name of an option usually is the one you find
+in the longform of the option, i.e. the name
+behind the ``--`` double-dash.
+
+IOW, you can set default values for options per project, per
+home-directoray, per shell session or per test-run.
+
Added: py/trunk/py/doc/test-dist.txt
==============================================================================
--- (empty file)
+++ py/trunk/py/doc/test-dist.txt Sun Mar 22 01:38:43 2009
@@ -0,0 +1,107 @@
+.. _`distribute tests across machines`:
+
+``py.test`` can ad-hoc distribute test runs to multiple CPUs or remote
+machines. This allows to speed up development or to use special resources
+of remote machines. Before running tests remotely, ``py.test`` efficiently
+synchronizes your program source code to the remote place. All test results
+are reported back and displayed to your local test session. You may
+specify different Python versions and interpreters.
+
+Speed up test runs by sending tests to multiple CPUs
+----------------------------------------------------------
+
+To send tests to multiple CPUs, type::
+
+ py.test -n NUM
+
+Especially for longer running tests or tests requiring
+a lot of IO this can lead to considerable speed ups.
+
+Test on a different python interpreter
+----------------------------------------------------------
+
+To send tests to a python2.4 process, you may type::
+
+ py.test --tx popen//python=python2.4
+
+This will start a subprocess which is run with the "python2.4"
+Python interpreter, found in your system binary lookup path.
+
+.. For convenience you may prepend ``3*`` to create three sub processes.
+
+
+Sending tests to remote SSH accounts
+------------------------------------
+
+Suppose you have a package ``mypkg`` which contains some
+tests that you can successfully run locally. And you
+have a ssh-reachable machine ``myhost``. Then
+you can ad-hoc distribute your tests by typing::
+
+ py.test -d --tx ssh=myhostpopen --rsyncdir mypkg mypkg
+
+This will synchronize your ``mypkg`` package directory
+to an remote ssh account and then locally collect tests
+and send them to remote places for execution.
+
+You can specify multiple ``--rsyncdir`` directories
+to be sent to the remote side.
+
+Sending tests to remote Socket Servers
+----------------------------------------
+
+Download the single-module `socketserver.py`_ Python program
+and run it on the specified hosts like this::
+
+ python socketserver.py
+
+It will tell you that it starts listening. You can now
+on your home machine specify this new socket host
+with something like this::
+
+ py.test -d --tx socket=192.168.1.102:8888 --rsyncdir mypkg mypkg
+
+no remote installation requirements
+++++++++++++++++++++++++++++++++++++++++++++
+
+Synchronisation and running of tests only requires
+a bare Python installation on the remote side. No
+special software is installed - this is realized through the
+*zero installation* principle that the underlying
+`py.execnet`_ mechanisms implements.
+
+
+.. _`socketserver.py`: ../execnet/script/socketserver.py
+.. _`py.execnet`: execnet.html
+
+Differences from local tests
+----------------------------
+
+* Test order is rather random (instead of in file order).
+* the test process may hang due to network problems
+* you may not reference files outside of rsynced directory structures
+
+Specifying test exec environments in a conftest.py
+-------------------------------------------------------------
+
+Instead of specifying command line options, you can
+put options values in a ``conftest.py`` file like this::
+
+ pytest_option_tx = ['ssh=myhost//python=python2.5', 'popen//python=python2.5']
+ pytest_option_dist = True
+
+Any commandline ``--tx`` specifictions will add to the list of available execution
+environments.
+
+Specifying "rsync" dirs in a conftest.py
+-------------------------------------------------------------
+
+In your ``mypkg/conftest.py`` you may specify directories to synchronise
+or to exclude::
+
+ rsyncdirs = ['.', '../plugins']
+ rsyncignore = ['_cache']
+
+These directory specifications are relative to the directory
+where the ``conftest.py`` is found.
+
Added: py/trunk/py/doc/test-examples.txt
==============================================================================
--- (empty file)
+++ py/trunk/py/doc/test-examples.txt Sun Mar 22 01:38:43 2009
@@ -0,0 +1,54 @@
+
+Working Examples
+================
+
+managing state at module, class and method level
+------------------------------------------------------------
+
+Here is a working example for what goes on when you setup modules,
+classes and methods::
+
+ # [[from py/documentation/example/pytest/test_setup_flow_example.py]]
+
+ def setup_module(module):
+ module.TestStateFullThing.classcount = 0
+
+ class TestStateFullThing:
+ def setup_class(cls):
+ cls.classcount += 1
+
+ def teardown_class(cls):
+ cls.classcount -= 1
+
+ def setup_method(self, method):
+ self.id = eval(method.func_name[5:])
+
+ def test_42(self):
+ assert self.classcount == 1
+ assert self.id == 42
+
+ def test_23(self):
+ assert self.classcount == 1
+ assert self.id == 23
+
+ def teardown_module(module):
+ assert module.TestStateFullThing.classcount == 0
+
+For this example the control flow happens as follows::
+
+ import test_setup_flow_example
+ setup_module(test_setup_flow_example)
+ setup_class(TestStateFullThing)
+ instance = TestStateFullThing()
+ setup_method(instance, instance.test_42)
+ instance.test_42()
+ setup_method(instance, instance.test_23)
+ instance.test_23()
+ teardown_class(TestStateFullThing)
+ teardown_module(test_setup_flow_example)
+
+Note that ``setup_class(TestStateFullThing)`` is called and not
+``TestStateFullThing.setup_class()`` which would require you
+to insert ``setup_class = classmethod(setup_class)`` to make
+your setup function callable. Did we mention that lazyness
+is a virtue?
Added: py/trunk/py/doc/test-ext.txt
==============================================================================
--- (empty file)
+++ py/trunk/py/doc/test-ext.txt Sun Mar 22 01:38:43 2009
@@ -0,0 +1,61 @@
+
+
+Learning by examples
+------------------------
+
+XXX
+
+adding custom options
++++++++++++++++++++++++
+
+py.test supports adding of standard optparse_ Options.
+A plugin may implement the ``addoption`` hook for registering
+custom options::
+
+ class ConftestPlugin:
+ def pytest_addoption(self, parser):
+ parser.addoption("-M", "--myopt", action="store",
+ help="specify string to set myopt")
+
+ def pytest_configure(self, config):
+ if config.option.myopt:
+ # do action based on option value
+
+.. _optparse: http://docs.python.org/library/optparse.html
+
+Setting default values for test options
+-----------------------------------------
+
+You can see all available command line options by running::
+
+ py.test -h
+
+py.test will lookup values of options in this order:
+
+* option value supplied at command line
+* content of environment variable ``PYTEST_OPTION_NAME=...``
+* ``name = ...`` setting in the nearest ``conftest.py`` file.
+
+The name of an option usually is the one you find
+in the longform of the option, i.e. the name
+behind the ``--`` double-dash.
+
+IOW, you can set default values for options per project, per
+home-directoray, per shell session or per test-run.
+
+
+
+Plugin methods
+----------------------------------
+
+A Plugin class may implement the following attributes and methods:
+
+XXX
+
+_`pytest event`:
+
+Pytest Events
+-------------------
+
+XXX
+
Added: py/trunk/py/doc/test-features.txt
==============================================================================
--- (empty file)
+++ py/trunk/py/doc/test-features.txt Sun Mar 22 01:38:43 2009
@@ -0,0 +1,246 @@
+
+Basic Features of ``py.test``
+=============================
+
+automatic collection of tests on all levels
+-------------------------------------------
+
+The automated test collection process walks the current
+directory (or the directory given as a command line argument)
+and all its subdirectories and collects python modules with a
+leading ``test_`` or trailing ``_test`` filename. From each
+test module every function with a leading ``test_`` or class with
+a leading ``Test`` name is collected. The collecting process can
+be customized at directory, module or class level. (see
+`collection process`_ for some implementation details).
+
+.. _`generative tests`:
+.. _`collection process`: impl-test.html#collection-process
+
+assert with the ``assert`` statement
+------------------------------------
+
+``py.test`` allows to use the standard python
+``assert statement`` for verifying expectations
+and values in Python tests. For example, you can
+write the following in your tests::
+
+ assert hasattr(x, 'attribute')
+
+to state that your object has a certain ``attribute``. In case this
+assertion fails you will see the value of ``x``. Intermediate
+values are computed by executing the assert expression a second time.
+If you execute code with side effects, e.g. read from a file like this::
+
+ assert f.read() != '...'
+
+then you may get a warning from pytest if that assertions
+first failed and then succeeded.
+
+asserting expected exceptions
+----------------------------------------------
+
+In order to write assertions about exceptions, you use
+one of two forms::
+
+ py.test.raises(Exception, func, *args, **kwargs)
+ py.test.raises(Exception, "func(*args, **kwargs)")
+
+both of which execute the given function with args and kwargs and
+asserts that the given ``Exception`` is raised. The reporter will
+provide you with helpful output in case of failures such as *no
+exception* or *wrong exception*.
+
+dynamically skipping tests
+----------------------------------------
+
+If you want to skip tests you can use ``py.test.skip`` within
+test or setup functions. Example::
+
+ py.test.skip("message")
+
+You can also use a helper to skip on a failing import::
+
+ docutils = py.test.importorskip("docutils")
+
+or to skip if a library does not have the right version::
+
+ docutils = py.test.importorskip("docutils", minversion="0.3")
+
+The version will be read from the module's ``__version__`` attribute.
+
+
+generative tests: yielding more tests
+-------------------------------------
+
+*Generative tests* are test methods that are *generator functions* which
+``yield`` callables and their arguments. This is most useful for running a
+test function multiple times against different parameters. Example::
+
+ def test_generative():
+ for x in (42,17,49):
+ yield check, x
+
+ def check(arg):
+ assert arg % 7 == 0 # second generated tests fails!
+
+Note that ``test_generative()`` will cause three tests
+to get run, notably ``check(42)``, ``check(17)`` and ``check(49)``
+of which the middle one will obviously fail.
+
+To make it easier to distinguish the generated tests it is possible to specify an explicit name for them, like for example::
+
+ def test_generative():
+ for x in (42,17,49):
+ yield "case %d" % x, check, x
+
+
+.. _`selection by keyword`:
+
+selecting/unselecting tests by keyword
+---------------------------------------------
+
+Pytest's keyword mechanism provides a powerful way to
+group and selectively run tests in your test code base.
+You can selectively run tests by specifiying a keyword
+on the command line. Examples:
+
+ py.test -k test_simple
+ py.test -k "-test_simple"
+
+will run all tests matching (or not matching) the
+"test_simple" keyword. Note that you need to quote
+the keyword if "-" is recognized as an indicator
+for a commandline option. Lastly, you may use
+
+ py.test. -k "test_simple:"
+
+which will run all tests after the expression has *matched once*, i.e.
+all tests that are seen after a test that matches the "test_simple"
+keyword.
+
+By default, all filename parts and
+class/function names of a test function are put into the set
+of keywords for a given test. You may specify additional
+kewords like this::
+
+ @py.test.mark(webtest=True)
+ def test_send_http():
+ ...
+
+testing with multiple python versions / executables
+---------------------------------------------------
+
+With ``--tx EXECUTABLE`` you can specify a python
+executable (e.g. ``python2.2``) with which the tests
+will be executed.
+
+
+testing starts immediately
+--------------------------
+
+Testing starts as soon as the first ``test item``
+is collected. The collection process is iterative
+and does not need to complete before your first
+test items are executed.
+
+support for modules containing tests
+--------------------------------------
+
+As ``py.test`` operates as a separate cmdline
+tool you can easily have a command line utility and
+some tests in the same file.
+
+debug with the ``print`` statement
+----------------------------------
+
+By default, ``py.test`` catches text written to stdout/stderr during
+the execution of each individual test. This output will only be
+displayed however if the test fails; you will not see it
+otherwise. This allows you to put debugging print statements in your
+code without being overwhelmed by all the output that might be
+generated by tests that do not fail.
+
+Each failing test that produced output during the running of the test
+will have its output displayed in the ``recorded stdout`` section.
+
+The catching of stdout/stderr output can be disabled using the
+``--nocapture`` option to the ``py.test`` tool. Any output will
+in this case be displayed as soon as it is generated.
+
+test execution order
+--------------------------------
+
+Tests usually run in the order in which they appear in the files.
+However, tests should not rely on running one after another, as
+this prevents more advanced usages: running tests
+distributedly or selectively, or in "looponfailing" mode,
+will cause them to run in random order.
+
+useful tracebacks, recursion detection
+--------------------------------------
+
+A lot of care is taken to present nice tracebacks in case of test
+failure. Try::
+
+ py.test py/doc/example/pytest/failure_demo.py
+
+to see a variety of 17 tracebacks, each tailored to a different
+failure situation.
+
+``py.test`` uses the same order for presenting tracebacks as Python
+itself: the oldest function call is shown first, and the most recent call is
+shown last. A ``py.test`` reported traceback starts with your
+failing test function. If the maximum recursion depth has been
+exceeded during the running of a test, for instance because of
+infinite recursion, ``py.test`` will indicate where in the
+code the recursion was taking place. You can inhibit
+traceback "cutting" magic by supplying ``--fulltrace``.
+
+There is also the possibility of using ``--tb=short`` to get regular CPython
+tracebacks. Or you can use ``--tb=no`` to not show any tracebacks at all.
+
+no inheritance requirement
+--------------------------
+
+Test classes are recognized by their leading ``Test`` name. Unlike
+``unitest.py``, you don't need to inherit from some base class to make
+them be found by the test runner. Besides being easier, it also allows
+you to write test classes that subclass from application level
+classes.
+
+disabling a test class
+----------------------
+
+If you want to disable a complete test class you
+can set the class-level attribute ``disabled``.
+For example, in order to avoid running some tests on Win32::
+
+ class TestPosixOnly:
+ disabled = sys.platform == 'win32'
+
+ def test_xxx(self):
+ ...
+
+testing for deprecated APIs
+------------------------------
+
+In your tests you can use ``py.test.deprecated_call(func, *args, **kwargs)``
+to test that a particular function call triggers a DeprecationWarning.
+This is useful for testing phasing out of old APIs in your projects.
+
+doctest support
+-------------------
+
+If you want to integrate doctests, ``py.test`` now by default
+picks up files matching the ``test_*.txt`` or ``*_test.txt``
+patterns and processes them as text files containing doctests.
+This is an experimental feature and likely to change
+its implementation.
+
+
+.. _`reStructured Text`: http://docutils.sourceforge.net
+.. _`Python debugger`: http://docs.python.org/lib/module-pdb.html
+
+
+.. _nose: http://somethingaboutorange.com/mrl/projects/nose/
Modified: py/trunk/py/doc/test-plugins.txt
==============================================================================
--- py/trunk/py/doc/test-plugins.txt (original)
+++ py/trunk/py/doc/test-plugins.txt Sun Mar 22 01:38:43 2009
@@ -1,67 +1,56 @@
-pytest plugins
-==================
+Many of py.test's features are implemented as a plugin.
-specifying plugins for directories or test modules
----------------------------------------------------------
-
-py.test loads and configures plugins at tool startup and whenever
-it encounters new confest or test modules which
-contain a ``pytest_plugins`` definition. At tool
-startup the ``PYTEST_PLUGINS`` environment variable
-is considered as well.
-
-Example
-++++++++++
-
-If you create a ``conftest.py`` file with the following content::
+Available plugins
+-----------------------
- pytest_plugins = "pytest_plugin1", MyLocalPluginClass
+py.test has a number of default plugins. You can see which
+ones by specifying ``--trace=config``.
-then test execution within that directory can make use
-of the according instantiated plugins:
+* adding reporting facilities, examples:
+ pytest_terminal: default reporter for writing info to terminals
+ pytest_resultlog: log test results in machine-readable form to a file
+ pytest_eventlog: log all internal pytest events to a file
-* the module ``pytest_plugin1`` will be imported and
- and its contained `Plugin1`` class instantiated.
- A plugin module can put its dependencies into
- a "pytest_plugins" attribute at module level as well.
+* marking and reporting test specially
+ pytest_xfail: "expected to fail" test marker
-* the ``MyLocalPluginClass`` will be instantiated
- and added to the pluginmanager.
+* funcargs for advanced
+ pytest_tmpdir: provide temporary directories to test functions
+ pytest_plugintester: generic apichecks, support for functional plugin tests
+ pytest_pytester: support for testing py.test runs
+* extending test execution, e.g.
+ pytest_apigen: tracing values of function/method calls when running tests
-Plugin methods
-----------------------------------
-A Plugin class may implement the following attributes and methods:
+Loading plugins and specifying dependencies
+---------------------------------------------------------
-* pytest_cmdlineoptions: a list of optparse-style py.test.config.Option objects
-* pytest_configure(self, config): called after command line options have been parsed
-* pytest_unconfigure(self, config): called before the test process quits
-* pytest_event(self, event): called for each `pytest event`_
+py.test loads and configures plugins at tool startup:
-XXX reference APIcheck'ed full documentation
+* by reading the ``PYTEST_PLUGINS`` environment variable
+ and importing the comma-separated list of plugin names.
-_`pytest event`:
+* by loading all plugins specified via one or more ``-p name``
+ command line options.
-Pytest Events
--------------------
+* by loading all plugins specified via a ``pytest_plugins``
+ variable in ``conftest.py`` files or test modules.
-XXX Various reporting events.
+example: ensure a plugin is loaded
+++++++++++++++++++++++++++++++++++++
-Example plugins
------------------------
+If you create a ``conftest.py`` file with the following content::
-XXX here are a few existing plugins:
+ pytest_plugins = "pytest_myextension",
-* adding reporting facilities, e.g.
- pytest_terminal: default reporter for writing info to terminals
- pytest_resultlog: log test results in machine-readable form to a file
- pytest_eventlog: log all internal pytest events to a file
- pytest_xfail: "expected to fail" test marker
- pytest_tmpdir: provide temporary directories to test functions
- pytest_plugintester: generic apichecks, support for functional plugin tests
- pytest_pytester: support for testing py.test runs
+then all tests in that directory and below it will run with
+an instantiated "pytest_myextension". Here is how instantiation
+takes place:
+
+* the module ``pytest_extension`` will be imported and
+ and its contained `ExtensionPlugin`` class will
+ be instantiated. A plugin module may specify its
+ dependencies via another ``pytest_plugins`` definition.
-* extending test execution, e.g.
- pytest_apigen: tracing values of function/method calls when running tests
Added: py/trunk/py/doc/test-quickstart.txt
==============================================================================
--- (empty file)
+++ py/trunk/py/doc/test-quickstart.txt Sun Mar 22 01:38:43 2009
@@ -0,0 +1,58 @@
+.. _`setuptools installation`: http://pypi.python.org/pypi/setuptools
+
+
+Installing py.test
+-------------------------------
+
+This document assumes basic python knowledge. If you have a
+`setuptools installation`_, install ``py.test`` by typing::
+
+ easy_install -U py
+
+For alternative installation methods please see the download_ page.
+
+You should now have a ``py.test`` command line tool and can
+look at its documented cmdline options via this command::
+
+ py.test -h
+
+Writing and running a test
+---------------------------
+
+``py.test`` is the command line tool to run tests.
+Let's write a first test module by putting the following
+test function into a ``test_sample.py`` file::
+
+ # content of test_sample.py
+ def test_answer():
+ assert 42 == 43
+
+Now you can run the test by passing it as an argument::
+
+ py.test test_sample.py
+
+What does happen here? ``py.test`` looks for functions and
+methods in the module that start with ``test_``. It then
+executes those tests. Assertions about test outcomes are
+done via the standard ``assert`` statement.
+
+You can also use ``py.test`` to run all tests in a directory structure by
+invoking it without any arguments::
+
+ py.test
+
+This will automatically collect and run any Python module whose filenames
+start with ``test_`` or ends with ``_test`` from the directory and any
+subdirectories, starting with the current directory, and run them. Each
+Python test module is inspected for test methods starting with ``test_``.
+
+.. Organising your tests
+.. ---------------------------
+
+Please refer to `features`_ for a walk through the basic features.
+
+
+.. _download: download.html
+.. _features: test-features.html
+
+
Modified: py/trunk/py/doc/test.txt
==============================================================================
--- py/trunk/py/doc/test.txt (original)
+++ py/trunk/py/doc/test.txt Sun Mar 22 01:38:43 2009
@@ -1,651 +1,22 @@
-================================
-The ``py.test`` tool and library
-================================
+*py.test* is a tool for:
-.. contents::
-.. sectnum::
+* rapidly writing unit- and functional tests in Python
+* writing tests for non-python code and data
+* receiving useful reports on test failures
+* distributing tests to multiple CPUs and remote environments
+quickstart_: for getting started immediately.
-This document is about the *usage* of the ``py.test`` testing tool. There is
-also document describing the `implementation and the extending of py.test`_.
+features_: a walk through basic features and usage.
-.. _`implementation and the extending of py.test`: impl-test.html
+plugins_: using available plugins.
-starting point: ``py.test`` command line tool
-=============================================
+extend_: writing plugins and advanced configuration.
-We presume you have done an installation as per the
-download_ page after which you should be able to execute the
-'py.test' tool from a command line shell.
+`distributed testing`_ how to distribute test runs to other machines and platforms.
-``py.test`` is the command line tool to run tests. You can supply it
-with a Python test file (or directory) by passing it as an argument::
-
- py.test test_sample.py
-
-``py.test`` looks for any functions and methods in the module that
-start with with ``test_`` and will then run those methods. Assertions
-about test outcomes are done via the standard ``assert`` statement.
-
-This means you can write tests without any boilerplate::
-
- # content of test_sample.py
- def test_answer():
- assert 42 == 43
-
-You may have test functions and test methods, there is no
-need to subclass or to put tests into a class.
-You can also use ``py.test`` to run all tests in a directory structure by
-invoking it without any arguments::
-
- py.test
-
-This will automatically collect and run any Python module whose filenames
-start with ``test_`` or ends with ``_test`` from the directory and any
-subdirectories, starting with the current directory, and run them. Each
-Python test module is inspected for test methods starting with ``test_``.
-
-.. _download: download.html
-.. _features:
-
-Basic Features of ``py.test``
-=============================
-
-assert with the ``assert`` statement
-------------------------------------
-
-Writing assertions is very simple and this is one of py.test's
-most noticeable features, as you can use the ``assert``
-statement with arbitrary expressions. For example you can
-write the following in your tests::
-
- assert hasattr(x, 'attribute')
-
-to state that your object has a certain ``attribute``. In case this
-assertion fails the test ``reporter`` will provide you with a very
-helpful analysis and a clean traceback.
-
-
-how to write assertions about exceptions
-----------------------------------------
-
-In order to write assertions about exceptions, you use
-one of two forms::
-
- py.test.raises(Exception, func, *args, **kwargs)
- py.test.raises(Exception, "func(*args, **kwargs)")
-
-both of which execute the given function with args and kwargs and
-asserts that the given ``Exception`` is raised. The reporter will
-provide you with helpful output in case of failures such as *no
-exception* or *wrong exception*.
-
-Skipping tests
-----------------------------------------
-
-If you want to skip tests you can use ``py.test.skip`` within
-test or setup functions. Example::
-
- py.test.skip("message")
-
-You can also use a helper to skip on a failing import::
-
- docutils = py.test.importorskip("docutils")
-
-or to skip if the library does not have the right version::
-
- docutils = py.test.importorskip("docutils", minversion="0.3")
-
-automatic collection of tests on all levels
--------------------------------------------
-
-The automated test collection process walks the current
-directory (or the directory given as a command line argument)
-and all its subdirectories and collects python modules with a
-leading ``test_`` or trailing ``_test`` filename. From each
-test module every function with a leading ``test_`` or class with
-a leading ``Test`` name is collected. The collecting process can
-be customized at directory, module or class level. (see
-`collection process`_ for some implementation details).
-
-.. _`generative tests`:
-.. _`collection process`: impl-test.html#collection-process
-
-generative tests: yielding more tests
--------------------------------------
-
-*Generative tests* are test methods that are *generator functions* which
-``yield`` callables and their arguments. This is most useful for running a
-test function multiple times against different parameters.
-Example::
-
- def test_generative():
- for x in (42,17,49):
- yield check, x
-
- def check(arg):
- assert arg % 7 == 0 # second generated tests fails!
-
-Note that ``test_generative()`` will cause three tests
-to get run, notably ``check(42)``, ``check(17)`` and ``check(49)``
-of which the middle one will obviously fail.
-
-To make it easier to distinguish the generated tests it is possible to specify an explicit name for them, like for example::
-
- def test_generative():
- for x in (42,17,49):
- yield "case %d" % x, check, x
-
-
-.. _`selection by keyword`:
-
-selecting/unselecting tests by keyword
----------------------------------------------
-
-Pytest's keyword mechanism provides a powerful way to
-group and selectively run tests in your test code base.
-You can selectively run tests by specifiying a keyword
-on the command line. Examples:
-
- py.test -k test_simple
- py.test -k "-test_simple"
-
-will run all tests matching (or not matching) the
-"test_simple" keyword. Note that you need to quote
-the keyword if "-" is recognized as an indicator
-for a commandline option. Lastly, you may use
-
- py.test. -k "test_simple:"
-
-which will run all tests after the expression has *matched once*, i.e.
-all tests that are seen after a test that matches the "test_simple"
-keyword.
-
-By default, all filename parts and
-class/function names of a test function are put into the set
-of keywords for a given test. You may specify additional
-kewords like this::
-
- @py.test.mark(webtest=True)
- def test_send_http():
- ...
-
-testing with multiple python versions / executables
----------------------------------------------------
-
-With ``--exec=EXECUTABLE`` you can specify a python
-executable (e.g. ``python2.2``) with which the tests
-will be executed.
-
-
-testing starts immediately
---------------------------
-
-Testing starts as soon as the first ``test item``
-is collected. The collection process is iterative
-and does not need to complete before your first
-test items are executed.
-
-no interference with cmdline utilities
---------------------------------------
-
-As ``py.test`` mainly operates as a separate cmdline
-tool you can easily have a command line utility and
-some tests in the same file.
-
-debug with the ``print`` statement
-----------------------------------
-
-By default, ``py.test`` catches text written to stdout/stderr during
-the execution of each individual test. This output will only be
-displayed however if the test fails; you will not see it
-otherwise. This allows you to put debugging print statements in your
-code without being overwhelmed by all the output that might be
-generated by tests that do not fail.
-
-Each failing test that produced output during the running of the test
-will have its output displayed in the ``recorded stdout`` section.
-
-The catching of stdout/stderr output can be disabled using the
-``--nocapture`` option to the ``py.test`` tool. Any output will
-in this case be displayed as soon as it is generated.
-
-test execution order
---------------------------------
-
-Tests usually run in the order in which they appear in the files.
-However, tests should not rely on running one after another, as
-this prevents more advanced usages: running tests
-distributedly or selectively, or in "looponfailing" mode,
-will cause them to run in random order.
-
-useful tracebacks, recursion detection
---------------------------------------
-
-A lot of care is taken to present nice tracebacks in case of test
-failure. Try::
-
- py.test py/documentation/example/pytest/failure_demo.py
-
-to see a variety of 17 tracebacks, each tailored to a different
-failure situation.
-
-``py.test`` uses the same order for presenting tracebacks as Python
-itself: the oldest function call is shown first, and the most recent call is
-shown last. A ``py.test`` reported traceback starts with your
-failing test function. If the maximum recursion depth has been
-exceeded during the running of a test, for instance because of
-infinite recursion, ``py.test`` will indicate where in the
-code the recursion was taking place. You can inhibit
-traceback "cutting" magic by supplying ``--fulltrace``.
-
-There is also the possibility of using ``--tb=short`` to get regular CPython
-tracebacks. Or you can use ``--tb=no`` to not show any tracebacks at all.
-
-no inheritance requirement
---------------------------
-
-Test classes are recognized by their leading ``Test`` name. Unlike
-``unitest.py``, you don't need to inherit from some base class to make
-them be found by the test runner. Besides being easier, it also allows
-you to write test classes that subclass from application level
-classes.
-
-disabling a test class
-----------------------
-
-If you want to disable a complete test class you
-can set the class-level attribute ``disabled``.
-For example, in order to avoid running some tests on Win32::
-
- class TestEgSomePosixStuff:
- disabled = sys.platform == 'win32'
-
- def test_xxx(self):
- ...
-
-testing for deprecated APIs
-------------------------------
-
-In your tests you can use ``py.test.deprecated_call(func, *args, **kwargs)``
-to test that a particular function call triggers a DeprecationWarning.
-This is useful for testing phasing out of old APIs in your projects.
-
-Managing test state across test modules, classes and methods
-------------------------------------------------------------
-
-Often you want to create some files, database connections or other
-state in order to run tests in a certain environment. With
-``py.test`` there are three scopes for which you can provide hooks to
-manage such state. Again, ``py.test`` will detect these hooks in
-modules on a name basis. The following module-level hooks will
-automatically be called by the session::
-
- def setup_module(module):
- """ setup up any state specific to the execution
- of the given module.
- """
-
- def teardown_module(module):
- """ teardown any state that was previously setup
- with a setup_module method.
- """
-
-The following hooks are available for test classes::
-
- def setup_class(cls):
- """ setup up any state specific to the execution
- of the given class (which usually contains tests).
- """
-
- def teardown_class(cls):
- """ teardown any state that was previously setup
- with a call to setup_class.
- """
-
- def setup_method(self, method):
- """ setup up any state tied to the execution of the given
- method in a class. setup_method is invoked for every
- test method of a class.
- """
-
- def teardown_method(self, method):
- """ teardown any state that was previously setup
- with a setup_method call.
- """
-
-The last two hooks, ``setup_method`` and ``teardown_method``, are
-equivalent to ``setUp`` and ``tearDown`` in the Python standard
-library's ``unitest`` module.
-
-All setup/teardown methods are optional. You could have a
-``setup_module`` but no ``teardown_module`` and the other way round.
-
-Note that while the test session guarantees that for every ``setup`` a
-corresponding ``teardown`` will be invoked (if it exists) it does
-*not* guarantee that any ``setup`` is called only happens once. For
-example, the session might decide to call the ``setup_module`` /
-``teardown_module`` pair more than once during the execution of a test
-module.
-
-Experimental doctest support
-------------------------------------------------------------
-
-If you want to integrate doctests, ``py.test`` now by default
-picks up files matching the ``test_*.txt`` or ``*_test.txt``
-patterns and processes them as text files containing doctests.
-This is an experimental feature and likely to change
-its implementation.
-
-Working Examples
-================
-
-Example for managing state at module, class and method level
-------------------------------------------------------------
-
-Here is a working example for what goes on when you setup modules,
-classes and methods::
-
- # [[from py/documentation/example/pytest/test_setup_flow_example.py]]
-
- def setup_module(module):
- module.TestStateFullThing.classcount = 0
-
- class TestStateFullThing:
- def setup_class(cls):
- cls.classcount += 1
-
- def teardown_class(cls):
- cls.classcount -= 1
-
- def setup_method(self, method):
- self.id = eval(method.func_name[5:])
-
- def test_42(self):
- assert self.classcount == 1
- assert self.id == 42
-
- def test_23(self):
- assert self.classcount == 1
- assert self.id == 23
-
- def teardown_module(module):
- assert module.TestStateFullThing.classcount == 0
-
-For this example the control flow happens as follows::
-
- import test_setup_flow_example
- setup_module(test_setup_flow_example)
- setup_class(TestStateFullThing)
- instance = TestStateFullThing()
- setup_method(instance, instance.test_42)
- instance.test_42()
- setup_method(instance, instance.test_23)
- instance.test_23()
- teardown_class(TestStateFullThing)
- teardown_module(test_setup_flow_example)
-
-
-Note that ``setup_class(TestStateFullThing)`` is called and not
-``TestStateFullThing.setup_class()`` which would require you
-to insert ``setup_class = classmethod(setup_class)`` to make
-your setup function callable. Did we mention that lazyness
-is a virtue?
-
-Some ``py.test`` command-line options
-=====================================
-
-Regular options
----------------
-
-``-v, --verbose``
- Increase verbosity. This shows a test per line while running and also
- shows the traceback after interrupting the test run with Ctrl-C.
-
-
-``-x, --exitfirst``
- exit instantly on the first error or the first failed test.
-
-
-``-s, --nocapture``
- disable catching of sys.stdout/stderr output.
-
-
-``-k KEYWORD``
- only run test items matching the given keyword expression. You can also add
- use ``-k -KEYWORD`` to exlude tests from being run. The keyword is matched
- against filename, test class name, method name.
-
-
-``-l, --showlocals``
- show locals in tracebacks: for every frame in the traceback, show the values
- of the local variables.
-
-
-``--pdb``
- drop into pdb (the `Python debugger`_) on exceptions. If the debugger is
- quitted, the next test is run. This implies ``-s``.
-
-
-``--tb=TBSTYLE``
- traceback verboseness: ``long`` is the default, ``short`` are the normal
- Python tracebacks, ``no`` omits tracebacks completely.
-
-
-``--fulltrace``
- Don't cut any tracebacks. The default is to leave out frames if an infinite
- recursion is detected.
-
-
-``--nomagic``
- Refrain from using magic as much as possible. This can be useful if you are
- suspicious that ``py.test`` somehow interferes with your program in
- unintended ways (if this is the case, please contact us!).
-
-
-``--collectonly``
- Only collect tests, don't execute them.
-
-
-``--traceconfig``
- trace considerations of conftest.py files. Useful when you have various
- conftest.py files around and are unsure about their interaction.
-
-``-f, --looponfailing``
- Loop on failing test set. This is a feature you can use when you are trying
- to fix a number of failing tests: First all the tests are being run. If a
- number of tests are failing, these are run repeatedly afterwards. Every
- repetition is started once a file below the directory that you started
- testing for is changed. If one of the previously failing tests now passes,
- it is removed from the test set.
-
-``--exec=EXECUTABLE``
- Python executable to run the tests with. Useful for testing on different
- versions of Python.
-
-
-
-experimental options
---------------------
-
-**Note**: these options could change in the future.
-
-
-``-d, --dist``
- ad-hoc `distribute tests across machines`_ (requires conftest settings)
-
-
-``-w, --startserver``
- starts local web server for displaying test progress.
-
-
-``-r, --runbrowser``
- Run browser (implies --startserver).
-
-
-``--boxed``
- Use boxed tests: run each test in an external process. Very useful for testing
- things that occasionally segfault (since normally the segfault then would
- stop the whole test process).
-
-``--rest``
- `reStructured Text`_ output reporting.
-
-
-.. _`reStructured Text`: http://docutils.sourceforge.net
-.. _`Python debugger`: http://docs.python.org/lib/module-pdb.html
-
-
-.. _`distribute tests across machines`:
-
-
-Automated Distributed Testing
-==================================
-
-If you have a project with a large number of tests, and you have
-machines accessible through SSH, ``py.test`` can distribute
-tests across the machines. It does not require any particular
-installation on the remote machine sides as it uses `py.execnet`_
-mechanisms to distribute execution. Using distributed testing
-can speed up your development process considerably and it
-may also be useful where you need to use a remote server
-that has more resources (e.g. RAM/diskspace) than your
-local machine.
-
-*WARNING*: support for distributed testing is experimental,
-its mechanics and configuration options may change without
-prior notice. Particularly, not all reporting features
-of the in-process py.test have been integrated into
-the distributed testing approach.
-
-Requirements
-------------
-
-Local requirements:
-
-* ssh client
-* python
-
-requirements for remote machines:
-
-* ssh daemon running
-* ssh keys setup to allow login without a password
-* python
-* unix like machine (reliance on ``os.fork``)
-
-How to use it
------------------------
-
-When you issue ``py.test -d`` then your computer becomes
-the distributor of tests ("master") and will start collecting
-and distributing tests to several machines. The machines
-need to be specified in a ``conftest.py`` file.
-
-At start up, the master connects to each node using `py.execnet.SshGateway`_
-and *rsyncs* all specified python packages to all nodes.
-Then the master collects all of the tests and immediately sends test item
-descriptions to its connected nodes. Each node has a local queue of tests
-to run and begins to execute the tests, following the setup and teardown
-semantics. The test are distributed at function and method level.
-When a test run on a node is completed it reports back the result
-to the master.
-
-The master can run one of three reporters to process the events
-from the testing nodes: command line, rest output and ajaxy web based.
-
-.. _`py.execnet`: execnet.html
-.. _`py.execnet.SshGateway`: execnet.html
-
-Differences from local tests
-----------------------------
-
-* Test order is rather random (instead of in file order).
-* the test process may hang due to network problems
-* you may not reference files outside of rsynced directory structures
-
-Configuration
--------------
-
-You must create a conftest.py in any parent directory above your tests.
-
-The options that you need to specify in that conftest.py file are:
-
-* `dist_hosts`: a required list of host specifications
-* `dist_rsync_roots` - a list of relative locations to copy to the remote machines.
-* `dist_rsync_ignore` - a list of relative locations to ignore for rsyncing
-* `dist_remotepython` - the remote python executable to run.
-* `dist_nicelevel` - process priority of remote nodes.
-* `dist_boxed` - will run each single test in a separate process
- (allowing to survive segfaults for example)
-* `dist_taskspernode` - Maximum number of tasks being queued to remote nodes
-
-Sample configuration::
-
- dist_hosts = ['localhost', 'user at someserver:/tmp/somedir']
- dist_rsync_roots = ['../pypy', '../py']
- dist_remotepython = 'python2.4'
- dist_nicelevel = 10
- dist_boxed = False
- dist_maxwait = 100
- dist_taskspernode = 10
-
-To use the browser-based reporter (with a nice AJAX interface) you have to tell
-``py.test`` to run a small server locally using the ``-w`` or ``--startserver``
-command line options. Afterwards you can point your browser to localhost:8000
-to see the progress of the testing.
-
-Development Notes
------------------
-
-Changing the behavior of the web based reporter requires `pypy`_ since the
-javascript is actually generated fom rpython source.
-
-.. _`pypy`: http://codespeak.net/pypy
-
-Future/Planned Features of py.test
-==================================
-
-integrating various test methods
--------------------------------------------
-
-There are various conftest.py's out there
-that do html-reports, ad-hoc distribute tests
-to windows machines or other fun stuff.
-These approaches should be offerred natively
-by py.test at some point (requires refactorings).
-In addition, performing special checks such
-as w3c-conformance tests or ReST checks
-should be offered from mainline py.test.
-
-more distributed testing
------------------------------------------
-
-We'd like to generalize and extend our ad-hoc
-distributed testing approach to allow for running
-on multiple platforms simultanously and selectively.
-The web reporter should learn to deal with driving
-complex multi-platform test runs and providing
-useful introspection and interactive debugging hooks.
-
-
-move to report event based architecture
---------------------------------------------
-
-To facilitate writing of custom reporters
-py.test is to learn to generate reporting events
-at all levels which a reporter can choose to
-interpret and present. The distributed testing
-approach already uses such an approach and
-we'd like to unify this with the default
-in-process py.test mode.
-
-
-see what other tools do currently (nose, etc.)
-----------------------------------------------------
-
-There are various tools out there, among them
-the nose_ clone. It's about time to look again
-at these and other tools, integrate interesting
-features and maybe collaborate on some issues.
-
-.. _nose: http://somethingaboutorange.com/mrl/projects/nose/
+.. _quickstart: test-quickstart.html
+.. _features: test-features.html
+.. _plugins: test-plugins.html
+.. _extend: test-ext.html
+.. _`distributed testing`: test-dist.html
More information about the pytest-commit
mailing list