[AstroPy] Testing Guidelines
Mark Sienkiewicz
sienkiew at stsci.edu
Mon Aug 15 11:29:45 EDT 2011
Erik Tollerud wrote:
> On Mon, Aug 1, 2011 at 11:03 AM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
>
>> If the decision goes that way, I can write up a document describing a common
>> subset that covers most of what you need to do in routine testing. Then you
>> make your standard "use the astrolib-defined nose/py.test subset whenever it
>> is practical". Only a few unusual cases will need to deviate.
>>
>
> That seems like a great idea to me.
I'll put something together when I get some time.
> Unless there are objections, I'm
> inclined to suggest we adopt py.test as the suggested runner based on
> it having the widest compatibility... Or are there still misgivings
> about this? (I'm speculating not given that there's been nothing on
> this topic in the last couple weeks.)
>
Unless somebody specifically requests a poll, I think we have a consensus.
>>> Except there's one problem: the configuration files are formatted
>>> differently... We want to use configuration settings for just one of
>>> the packages so that the suite always runs the same.
>>>
>> What configuration file is that? I don't recall ever writing a
>> configuration file for either nose or py.test.
>>
>
> In setup.cfg you can add a [nosetest] or a [pytest] section to
>
I see - you are talking about a setuptools config file, not a nose
config or py.test config. I'll have to read up on what that is about.
> Assuming we adopt py.test, though, the other useful change would be to
> generate the standalone py.test script and hook it into "python
> setup.py test" so that no one has to download py.test to run the test
> suite with py.test. That's the only thing I can see that would
> definitely require a source code change.
>
Yes. Put that in the coding standard. We should have a script (with a
standardized name) to update all the generated source code. It should
re-gen C interface defintions, and make the py.test standalone script.
Presumably there may be other things, e.g. if the author uses a parser
generator or something like that.
>> IMHO, the most interesting feature to me are the new comparison functions.
>>
>
> Just so we're on the same page, by this do you mean all the new
> assert* methods of the TestCase class?
>
Yes.
> It's not clear to me that it's possible to use these comparison tool
> outside unittest2 - they seem to all be methods of TestCase and thus
> have to be done inside a unittest TestCase object (which is the whole
> point of using nose or py.test to avoid). As you say, that's really
> too bad, because they have some nice features.
>
All you need is to make a TestCase object and call the comparison
methods. Here is a proof of concept:
import unittest2
class dummy_ut2( unittest2.TestCase ) : # not named test_something
__test__ = False # just in case
def runTest( wtf ) :
raise Exception('Should never be called')
xcmp = dummy_ut2()
def test_a() :
xcmp.assertAlmostEqual( 1.001, 1.002, places=4 )
In principle, you can bury this kind of thing in a library, then use the
comparison functions even when you don't want to make classes based on
unittest2.
The major drawback I see here is that py.test reports a complete listing
of the function that raises the exception, NOT the function that it
called to run the test. So, you get a listing of assertAlmostEqual
instead of a listing of test_a. I have in mind various ideas for
workarounds.
So why would you do this?
- get the nice comparison functions of unittest2 without re-implementing
them
- make them available to any test system
Of course, nothing here stops anybody from writing
assert a == b
when that is the convenient thing to do.
> It looks to me like most of this sort of useful contextual information
> is part of in py.test's output, though (see
> http://pytest.org/latest/example/reportingdemo.html#tbreportdemo).
> The multiline string diff as well as a sequence diff reporting seems
> to be builtin to py.test if you just do "assert a==b" ... I didn't
> realize that until now - pretty neat!
>
That is nice. I had not seen that before.
More information about the AstroPy
mailing list