[py-dev] utest thoughts

[holger krekel]

Hi Ian, hi all, 

[Ian Bicking]
> holger krekel wrote:
> >Having said this, for having multiple data sets it's probably best
> >to introduce a mechanism into the Module Collector to look for 
> >custom module-defined collectors which would seamlessly integrate 
> >into the collection process of Units.  By implementing your own 
> >collection function you can yield many Units with different 
> >test data sets which are then executed/accounted for individually. 
> 
> Could this be like:
> 
> data = [(1, 'one'), (2, 'two'), ...]
> def test_collector():
>     for args in data:
>         # Depending on when tests are run, I think leaving out args=args
>         # could silently make all your tests use the same data :(
>         yield lambda args=args: test_converter(*args)
> # Should we then to test_collector = test_collector() ?

yes, though 

a) the collection function should not have a 'test' pre- or postfix 
   i think.  

b) collectors always yield collectors or Units, not plain functions 
   so you would actually do 

      yield py.test.Unit(my_custom_test_function, *args)

In the current py code (see my other postings) you'll find 
in py/test/test/data/Collector.py the current way to do custom 
collectors.  This Collector class is instantiated with 
the module's location and takes over the collecting 
process for the module.  No further attempt is made 
to collect anything from a module which contains a 
custom "Collector" object. 

> In the case I'm thinking of, where I run the identical set of tests on 
> different backends, it should be available as a command-line argument. 
> But if there's a generic command-line argument (like -D) then that could 
> be used to set arbitrary options (assuming the config file can accept 
> arbitrary options).

you already took '-D' for the pdb debugging but I guess we don't need
"--pdb" as a short option. I took the freedom to rename it from 
"--usedpdb" by the way. 

Introducing "-D" for passing options/backends to the tests 
still requires more careful thoughts about test configuration
(files) in general. I think it should be possible to have 
test-configuration files per directory, which might modify 
the collection process and deal with configuration data ("-D") 
issues. They _may_ also provide a different runner if it 
turns out to make sense.  Again please note, that py.test's 
"runner" has simpler responsibilities than unittest.py - based 
runners. We have the runner, the collectors and the reporter. 
 
> >Note, that it should always be possible to run tests of any
> >application with py-tests by invoking 'py.test APP-DIRECTORY' or 
> >simply 'py.test' while being in the app directory.  
> 
> What about "python setup.py test" ?  This would allow for a common way 
> to invoke tests, regardless of runner; people could use this for their 
> unittest-based tests as well, or whatever they are using.

I think it's easier to say 

    py.test APP-DIRECTORY 

and i don't want to deal with distutils-hacks if it can be avoided ... 

> I think I tried this at one point, but then got bored of trying to 
> figure out the distutils just to add this one little command.

... we seem to share the same view here :-) 
 
> In Zope3 they explicitly add doctests to the testing, it isn't just 
> automatically picked up.

Argh! I have to say i dislike writing repetitive unneccessary
"manually-synced" boilerplate code for tests, be they unit,
doc or any other kind of test. 

> In part because the doctests are typically in modules that
> aren't otherwise inspected for tests (they are inline with
> the normal code, not in seperate test_* modules).  I think
> there may be a performance issue with inspecting all modules
> for doctests, and potentially an issue of finding things
> that look like tests but aren't (though that probably isn't
> a big problem, since there's ways to exclude docstrings from
> doctest).

Hum, performance problem.  I think importing all test and implementation 
code of the py lib requires 2 seconds on a file system. As 
the collection happens iterative these two seconds are distributed 
across the whole testing time.  With "Test-Session" modes it 
will be less. 

But if there is a performance problem we may think of ways 
to exclude files from looking for doctests by means of the
per-directory py.test file. 
 
> >>* Code coverage tracking.  This should be fairly straight-forward to add.
> >
> >yes.  I guess everybody uses sys.settrace which is
> >unfortunately rather expensive. Anyway, it would be nice to
> >come up with some real life use cases and see what is really
> >useful. I am not clear on that. 
> 
> I think the "50% code coverage" is mostly a feel-good measure, so you 
> can be pleased with your increasing score as you add tests.  It would be 
> awesome to allow for leveling up with your tests.  "20% code coverage; 
> you have begun your travels" or "95% code coverage; you are approaching 
> enlightenment".

yes, sounds nice. 
 
> The actual file-by-file reports are more useful, I think.  Coverage 
> should only be tracked when explicitly asked for.

Yes, i guess so.  The same is true for profiling. 

> >>* Different levels of tests (-a --at-level or --all; default level is 1, 
> >>which doesn't run all tests).  They have lots of tests, so I'm guessing 
> >>they like to avoid running tests which are unlikely to fail.
> >
> >having different levels of tests seems interesting. I'd like
> >more of a keyword based approach where all tests are
> >associated with some keywords and you can select tests by
> >providing keywords.  Keywords could be automatically
> >associated from filename and python name components e.g.
> >['std', 'path', 'local', 'test_listdir'].  You could
> >additionally associate a 'slow' keyword to some tests (somehow) 
> >and start 'py.test --exclude="slow"' or put this as a default in
> >the configuration file. 
> 
> That would work well, I think.  How might tests be annotated?  On a 
> module-by-module basis, using some particular symbol 
> (__test_keywords__)?  Function attributes?  On an ad hoc basis by 
> customizing the collector?

It has to be possible to add keywords (i think we should
only ever deal with adding keywords) on the module, class and 
method level.  Finding syntax for the module level and the 
class level boils down to thinking about a good name which 
lists the additional keywords. 

For the method level we would have to worry about syntax 
so i'd like to avoid it alltogether.   

I think that with "automatic keywords", derived from 
the name of the test module, class (if any) and method 
name we are all set. You can always "add" a keyword by
extending the testname, avoiding any redundancy. 

For example, we could introduce "bug" tests: 

    def test_bug001_pypath_sucks():
        # ... 

and then you could run a specific "bug001" test by something like 

    py.test --include="bug001"

note, btw, that i plan to distribute ".svn" directories along with 
every copy of 'py'. This allows _everyone_ commit access 
to test_*.py and *_test.py files in the repository! So if 
someone finds a problem or wants a guarantee from the py lib 
he can simply contribute it!  Committing to the implementation 
tree will require registration for an account, though. 

> It's actually the kind of place where adaptation would be interesting; 
> objects would be adapted to test cases, where part of the test case API 
> was a set of keywords.  That would allow for a lot of customization, 
> while the actual tests could remain fairly simple.  Part of the base of 
> py.test would be adapters for packages, modules, and functions; the 
> module adapter looks for the test_* functions, function adapters might 
> look for function attributes, etc.  There'd be another adapter for 
> unittest.TestCase and unittest.TestSuite, and so on.  Packages could 
> create their own adapters for further customization.

I am not sure, i am following.  What would be the distinct advantates
of introducing this machinery? 
 
> >    - armin and me want a mechanism by which to include '.c'
> >      files in the library and seemlessly compiling (and 
> >      possibly caching) them via distutils-mechanism.  This
> >      should allow to work with a svn-checkout containing 
> >      c-coded modules without any explicit user interaction 
> >      and especially without distutils-installing it. 
> 
> Right, this is what Zope is doing.  It builds the package (but does not 
> install it) before running the tests (python setup.py build).

We mean it a lot more automatic.  You should not need to run _any_ 
intermediate command, no matter how simple.   A checkout of the
py lib should be enough.  And you can simply modify your '.c' 
file somewhere under the implementation tree and nicely expose 
any objects implemented via the "package export" runtime mechanism. 

The motto of the py lib is "avoiding APIs" :-) 

> >    - managing the py library versions in the long run, i'd 
> >      like to have an easy automated way for users of the py lib
> >      to get/install a new version.  preferably via a 'py' binary
> >      which also allows to ask 'py --version', 'py --selftest' 
> >      to let the py-lib tests run, 'py --test' which would iterate 
> >      into all modules/packages to find tests and run them. This is 
> >      kind of integrity test for your system. Also  'py someprogram.py' 
> >      could start an (interactive) python interpreter allowing 
> >      the script to choose/restrict it's version (on all platforms). 
> >
> >    - eventually i'd like to think some more about the notion 
> >      of a 'py' application which would be installable/manageable 
> >      probably connecting to the PyPI project but including 
> >      downloads. 
> 
> Is there a reason these separate concerns go together?  The last seems 
> like a distutils enhancement.  Handling multiple versions... well, 
> that's another issue that is pretty much unaddressed at this point, but 
> I'm not sure

Well, we need not discuss this further right now, i guess.  I just wanted
to present some of the ideas i am having.  I do think that the notion
of a "py app" which works well with the py-commandline utilities, 
is installable and upgradeable seemlessly from remote places is
something to go for.  It would be more than distutils and probably
sitting on top of it and PyPI. 

> >>* Run tests in a loop (-L --loop).  Also for checking memory leaks. 
> >>I've thought that running loops of tests in separate threads could also 
> >>be a useful test, for code that actually was supposed to be used with 
> >>threads.  That might be another place for specializing the runner.
> >
> >Yes, see above. Maybe 'py.test --session' could actually not return 
> >until all tests pass and wait for changing files to order to try again. 
> >This is really nice because you can just save from your editor and 
> >see the tests running (at least if you are working in multiple windows 
> >or on Xinerama like i do :-) 
> 
> I think for a text reporter this would lead to information overload.  

Well, the tests would only run (and fail) when implementation files 
got modified.  So most of the time it would be sitting idly and
print nothing. I have used this scheme, it's extremely nice, 
and doesn't overload with information.  It just avoids typing
the same commands over and over. 

cheers, 

    holger