[py-dev] Integrating doctests
holger krekel
hpk at trillke.net
Thu May 12 13:46:16 CEST 2005
Hi Ian,
i have been away for a few days (and everyone seems to know this
and mails me like crazy). I have the impression you might have
want to split your mail into multiple ones, but maybe i am just
missing the red line somehow.
On Tue, May 10, 2005 at 18:04 -0500, Ian Bicking wrote:
> OK, so I really want to integrate doctests, along the lines I mention in
> this post:
>
> http://blog.ianbicking.org/building-testability-into-web-applications.html
Can't reach that server at the moment (not the first time, btw).
> So, here's what I have so far. There's a couple parts. Here's a
> function to test unit tests:
You started off with doctests, and now talk about unittests. Probably
your weblog entry would help understand me your line of thinking better ...
> def unittest_tester(test):
> result = test.defaultTestResult()
> test.run(result)
> for flavor, lst in [('FAIL', result.failures),
> ('ERROR', result.errors)]:
> for test, err in lst:
> print '%s: %s' % (flavor, test)
> print err
> if result.failures or result.errors:
> py.test.fail('Errors occurred')
>
> This is a little lame, but I haven't gotten far enough into py.test to
> figure out reporting.
There is definitely a missing py.test customization hook which
allows a user to customize reporting from e.g. a conftest.py
file or from simply invoking py.test.fail()
> Actually, I still feel totally mystified by py.test, I just
> wander around poking things until they stop breaking. Is
> there anyplace I should be starting at to figure out quite
> what is going on?
I at least wrote some documentation about the internal workings:
http://codespeak.net/py/current/doc/test.html#collecting-and-running-tests-implementation-remarks
(sorry for the long URL). Have you read that and deemed it unworthy
or unhelpful?
> It's kind of that Ravioli Code kind of feel; a description
> of the role of all the classes would be really useful.
> Though personally I like descriptions that follow the code
> path chronologically, it's a good compliment to the source
> which tends not to have many chronological hints.
I had the impression that i keep py-dev updated about refactorings
and the intentions and even provide some detail of what's going
on. But that can certainly be improved.
> Anyway, I needed that function because I'm using doctest's
> TestSuite-building capability, though it'd probably be just as easy to
> use doctest.DocTest directly. Anyway, I really think unittest testing
> is a useful feature that belong in there somewhere,
You are mixing unittests and doctests freely here. I admit i
still haven't looked to closely into doctest's internal workings
so do you mean to imply that doctest and unittests are very
uniformly handled with Python already?
> ... in addition to doctests, since I don't always have the
> ability to change other projects' test structure.
so here you mean that py.test should grok existing
(unit/doc-)test structures? I am not against the idea but it
is a long way to follow that road i am afraid and i am not
ready to follow that myself at the moment. (apart from the
fact that we have certain ways of running unittests/doctests
in PyPy with py.test mechanisms but it's code that i'd rather
not like to explain at the moment, because it jumps through
more hoops than neccessary: it additionally runs against
the PyPy interpreter instead of against CPython).
> I don't think the support has to be very sophisticated; fix
> the errors in that example and it might be good enough. If
> it doesn't integrate particularly well into other UIs, like
> Tkinter, I don't think that's a big deal.
I'd like Jan Balster's tkinter frontend to remain as much
usable as possible and although i break Jan's code from
time to time (sorry, Jan!) i'd like it very much to be
an alternative frontend.
> After that I have this stuff that traverses all modules and gets
> doctests from some of the modules:
>
> class Directory(py.test.collect.Directory):
> def filefilter(self, path):
> return (path.check(fnmatch="*.py")
> and path.purebasename != 'conftest')
>
> class Module(py.test.collect.Module):
>
> def buildname2items(self):
> if (self.name.startswith('test_')
> or self.name.endswith('_test')):
> d = super(Module, self).buildname2items()
> else:
> d = {}
> try:
> test_suite = doctest.DocTestSuite(
> self.obj, setUp=setup_doctest,
> optionflags=flags)
> except ValueError, e:
> if 'has no test' in str(e):
> return d
> else:
> raise
> for i, x in py.builtin.enumerate(unpack_testsuite(test_suite)):
> name = 'doctest[%d]' % i
> d[name] = self.Function(name, self, (x,), obj=unittest_tester)
> return d
>
>
> But this is kind of awkward -- I don't want to look for normal tests in
> non-test_*-named files, only doctests, so I have to undo the filefilter
> in Directory and then reapply it in Module. And I get lots of modules
> that have no tests in my reports, which wastes space.
Right. A better way would be to override the Directory collector
and produce two kinds of Modules, usual py.test ones and ones
with doctests with code like this (untested):
class Directory(py.test.collect.Directory):
def buildname2items(self):
# just let our base class do its building
d = super(Directory, self).buildname2items()
# let's look for doctests ...
for fn in self.fspath.listdir('*.py'):
if fn.basename in d or fn.basename == 'conftest.py':
continue
# we have a candidate for doctests
if not isdoctestmodule(fn): # if possible to implement that
continue
d[fn.basename] = DoctestModule(fn, parent=self)
class DoctestModule(py.test.collect.Module):
def run(self):
module = self.fspath.pyimport()
# setup module appropriately
try:
# run doctests from 'module'
finally:
# teardown module appropriately
Like said earlier, this DoctestModule should have a way to "take over"
reporting somehow. It doesn't at the moment and be warned that there
might be other slight issues. But basically the above should
work rather cleanly.
> * I'd like a way to halt all tests. With SQLObject your environment
> must be configured properly, since you have to access a database. It's
> overwhelming when this isn't the case, because you get a huge list of
> failing tests. I'd like to raise an error when that happens so that
> py.test will abort all future tests. Hmm... I haven't tried SystemExit
> or KeyboardInterrupt. Maybe that'd do on its own.
Have you tried py.test.exit(msg)?
cheers,
holger
P.S.: Please note that py.test is still under development and there are
sometimes reasons why things are not done yet: often i prefer to
refactor and consolidate the (not so small number of) features
before i head off adding yet more features.
More information about the Pytest-dev
mailing list