[py-dev] order of setup / teardown

[holger krekel]

Hi Philippe, 

(note to all: i am back from a long and fun vacation 
and will see to answer pending issues in the next days)

On Mon, Apr 12, 2010 at 12:37 +0200, Philippe Fremy wrote:
> Hi,
> 
> I am not sure if this has been brought before, but I am surprised to see
> some calls inside the test "hooks" after the test session has finished.
> 
> I am referring here to teardown_class, revealed by a quick example:
> 
> pytest_sessionstart
> 
> setup_class
> setup_method
> 
> test1
> 
> teardown_method
> 
> setup_method
> test2
> 
> teardown_method
> 
> pytest_sessionfinish
> 
> teardown_class
> 
> This is creating problems for me because
>          I am assuming that all resources are setup in sesssionstart,
> and torn down in sessionfinish. And teardown_class is using  one of
> those resources.
> 
> Is it a bug or intended behavior ?

setup and teardown of python test functions are implemented through the builtin
py/_plugin/pytest_runner.py plugin.  It itself implements hooks called from the
"session" in order to provide the setup/teardown semantics.  There, 
it only tears down class/module state if the "next" test does not need it and
this is decided at setup time of said next test.  This means that
after the last test has run, some resources might still require teardown
and currently this is done through implementing it in a pytest_sessionfinish 
hook. The latter, however, is invoked after all user-provided hooks have
been invoked. 

So much for the analysis.  Also related to your other "directory-setup" 
issue i am not entirely happy about pointing people to pytest_namespace
and sessionstart/sessionfinish for scoped python setup purposes. Maybe
more python-specific setup/teardown hook(s) would make more sense?

Relatedly, I ponder giving up on the incremental collection of tests and
internally immediately produce a fully exploded lists of tests at 
session start.  This way teardown could be done in a more precise manner --
right after the resource/scope is not needed anymore.  It would also make
randomization and other test re-ordering easy.  A downside is that it takes
longer for the first test to run.  A plus is that it probably eases
implementation and provides less surprises (such as the one you are having
because pytest_runner would not need to work through sessionfinish anymore).

If you don't tell anyone - :) - there also is a immediate work around 
to your issue i think.  You could implement something like

    def pytest_sessionfinish(__multicall__):
        __multicall__.execute()  # call other sessionfinish implementers first

'__multicall__' is the object representing the current
ongoing plugin hook call and allows to dynamically call other reamining
hook implementers first. It's not documented or advertised 
(but used sometimes from pytest's own builtin plugins) because i 
think a more declarative decorator-based way of setting call 
ordering is saner.  Then again, i hope to avoid the need 
altogether for simplicity sake. 

cheers,
holger