[py-dev] Backtracking on doctest

[Ian Bicking]

holger krekel wrote:
> On Fri, May 13, 2005 at 14:00 -0500, Ian Bicking wrote:
> 
>>Well, now I'm starting to backtrack.  After I extended my test while 
>>adding new features, I looked back over my doctests and realized they 
>>were pretty pointless.  For brevity, most of my code no longer produced 
>>output at all, but instead was doing internal inserts.  If I didn't have 
>>a wrapper around my application for testing this might not have been 
>>feasible, but without a wrapper the testing wouldn't be feasible.
> 
> 
> Interesting.  With PyPy we noticed that we have lots of problems
> with some doctest/output-comparison related CPython regression 
> tests.  Doctests and output-comparisons are pretty sensitive 
> to the exact way things are represented as strings.  Nevertheless
> i still like the general doctest idea. 

It's not too terribly hard to override the comparison that doctest does. 
  Personally I like fuzzier comparisons in general.  There's not a very 
good way to install those comparisons, so if py.test integrates doctest 
it would be nice to have a convention for that.

For instance, for HTML-generating code I've written code to do more 
semantic XML comparisons that ignore some whitespace and the order of 
attributes.

>>Anyway, I converted it to a normal py.test test.  It seems much more 
>>reasonable.  I'm a little sad that it's not sitting right next to the 
>>servlet that it tests -- I liked that the tests were in a docstring 
>>(though they were also becoming rather long-winded, so maybe that's not 
>>ultimately practical either).  
> 
> 
> Did you consider putting tests in the same source file as the code
> being tested?  It's probably often not too nice but i am using it
> for one-file scripts and simply invoke 'py.test myscript.py'. 

I had, but I'm not sure if it would help a lot.  I'd still need the 
hooks in conftest, because I'd wanted those tests to be automatically 
picked up.  If it's just a function at the bottom of the module, I don't 
think that's actually much better than a separate module.

I think I had this idea that I could put ten lines of doctest at the top 
of my module that would be this concise documentation and description of 
the module, plus a test.  I'm not sure that's feasible, though it is 
still possible to use that for basic exercising of the code -- basically 
a one-liner that tells what a valid URL would be.  And, maybe, with a 
subclass of the standard Paste fixture you could add things like:

   app.upload_csv([('Name', 'Email'), ('Bob', 'bob at company.com')])

And that could be high enough level to get some benefit from being in a 
docstring.  But those kinds of methods (like "upload_csv") would have to 
be implemented in a class specifically meant for testing a single 
screen; which is also a useful idea, but I haven't really gotten that 
far in generalizing this stuff.

>>setup_module turned out to be quite useful; I added this to my conftest, 
>>and do a "from conftest import setup_module" in my module:
>>
>>def setup_module(module):
>>    app = TestApp(server.make_app(CONFIG.current_conf()),
>>                  CONFIG.current_conf())
>>    module.app = app
>>    module.CONFIG = CONFIG
>>    fixture.reset_state()
> 
> 
> I guess we should start a "recipe" section in the
> documentation for nice ideas like this.    

Once I get this all figured out I'll probably document the entire 
process for Paste, and include this code somewhere in the project.

> Btw, i once thought about making your above
> conftest.setup_module(module) get invoked automagically just
> like Collectors/Items are looked up.  An advantage could be
> that you could stack setup_module initializations but i am not
> sure how useful that is (it might be for larger projects).
> It's easy to add this feature should we want it. 

I could speculate, but I should probably keep trying to implement tests 
and then figure out what's works and what feels awkward.  Since I only 
have one module tested so far, "boilerplate" in that one module is no 
worse than a line of code in conftest ;)

>>I think it's likely that other boilerplate stuff will go in there in the 
>>future.  I'd still like doctests to work nicely, but it becomes less 
>>important to me.
>>
>>Anyway, no requests here, just thought I'd share my current and always 
>>changing thoughts on the matter.
> 
> 
> Thanks a lot.  I admit that your "ravioli" term used earlier deeply 
> hurt my feelings until i realized that you didn't say Spaghetti :-) 

Heh, sorry ;)  It's a term I came upon one day on the wiki 
(http://c2.com/cgi/wiki?RavioliCode) and it kind of stuck in my mind as 
a useful metaphor, but I forget that no one else knows what I mean.  If 
I had read up and knew what method to start looking at, I'd probably 
wouldn't have felt that way -- though it's always a risk with 
dynamically typed code, because in isolation it can be hard to tell what 
object you are interacting with.  For instance, I had a hard time 
figuring out when a variable was the object itself, or its collection 
wrapper (e.g., collect.Module or the actual module object).

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org