[py-dev] Exceptions in setup/teardown hooks

[holger krekel]

Hi Matthew, 

On Mon, Aug 11, 2008 at 08:17 +1200, Matthew Edwards wrote:
> Hi,
> 
> I've got some tests for a module which manages log files. The setup_class
> and teardown_class functions create and delete a directory for them, and the
> teardown_method function deletes the individual files based on a list. When
> a test fails, it can leave files which aren't on the list, so don't get
> deleted in teardown_method, which makes teardown_class choke on os.rmdir
> (because the directory is not empty). While I do need to know if there are
> files left over, the problem is that when an exception occurs in a hook, it
> shows a traceback for that only, and not the actual test failure, so I can't
> tell what failed.

Hum, I think failing setup/teardown methods could use more helpful
debug info.  This should be better on trunk soon (after i
merge the branch i am currently working on).  

Otherwise i agree with Neil.  setup/teardowns should not
contain test code by design. In practise they often 
call into application code that may fail and there is
nothing wrong with being more helpful about this.  

Concretely, I think a "self.assert_nofilesleft()" at the end
of the tests in question is fine and makes it obvious what is
expected program behaviour. 

> Am I just doing this wrong, or is this a bit of a bug? I would prefer an
> exception in [setup|teardown]_method to fail that method, and an exception
> in [setup|teardown]_class to fail all the tests in that class, and if an
> exception is raised in a test and the following hook, tracebacks for both
> should be displayed.

I agree that teardown_method/teardown_function reporting should 
closely relate to the test object.  I don't think that 
teardown_class should fail all tests retroactively - they were
already reported as success.  Again, it should indeed get
easier to see what test class failed. 

> As an aside, stdout capture doesn't work particularly well from the hooks
> either.

is better already. 

cheers,

holger