[py-dev] py.test and --session, re-run failures only

[holger krekel]

Hi Martin, 

nice to have you here!  I hope you had a good flight back to Canada. 

On Thu, Mar 31, 2005 at 21:23 -0500, Martin Blais wrote:
> hi guys
> 
> my first post on this list.  thanks for py.test, it's very nice, i'm
> in the process of converting my cr*ppy unittest hacks into something a
> little cleaner by using py.test.  in the past, i wrote some testing
> infrastructure for a large C++ project in the past, i hope i can
> contribute something to this project in the future.

sounds good! 

> 1.
> so i'm checking out py.test and i run across this:
> 
>     --session          run a test session/rerun only failing tests. 
> 
> "rerun only failing tests" lets me think  that somehow the session is
> recording somewhere (e.g. in a file) which tests have failed and only
> reruns those tests.  in fact, fiddling in the code and running it, i
> realized that it just runs a subprocess and monitors file changes (by
> polling), and then automatically reruns the tests.  perhaps the help
> string should be clarified.

Yes, although it's not exactly easy to put this in a short help text. 
If you have a suggestion please send it. 

> in any case, it would be a cool feature to be able to record which
> tests fail and then to be able to rerun just the ones which failed in
> the past.  we used to have that in my previous job, and it was very
> useful, we had LOTS of tests, and running them was very slow, so just
> rerunniing the failures was a great timesaver.

Yes, recording failing tests is a good idea.  The problem is determining 

    a) the location to store failing test information  
    b) the format in which "test addresses" should be stored 

a) points to a more general problem (having per-user storage for the py
lib and py.test in particular) that i'd like to solve in the near future. 
b) is more or less handlded in the "collect" branch which i will need to
merge real soon now.  Basically a "test address" consists of 
a rootpath and a list of names.  It's important that such "addresses" 
will work nice for the upcoming doctest support. 

> 2.
> i would like to be able to say "run just this one test", by specifying
> the method, e.g. test_search.py:TestSearch.test_simple, or perhaps
> some kind simpler pattern matching.  this is why i started fiddling in
> the code.

Very valid use case and i have a plan how to satisfy it ... 

> i would probably modify the driver to filter down the wanted methods
> once the collector has passed.  any comments on how to go about this
> welcome.

... first of all, note that the collect branch contains the future version
of py.test.  Second, i have added a chapter in the py.test documentation 
that describes future features (still open to discussion). Have a look at: 

http://codespeak.net/py/current/doc/test.html#future-planned-features-of-py-test

cheers, 

    holger