[pypy-dev] Improving the documentation on how we test

[Eli Stevens (Gmail)]

It's not clear if you're looking for input from the peanut gallery,
but here is my $0.02:

First test at the level of the feature as exposed to a user. This
makes sense to write before implementation starts, and is highly
unlikely to change if the implementation gets refactored. It sounds
like your integration test is what I mean, though I usually think of
it as even higher "feature level" testing.

I typically write unit tests interleaved with implementation, and only
to verify aspects of the implementation that aren't trivial to
desk-check. These will almost certainly get thrown away during a
refactor, so I try to not over-invest in them.

Finally, I try to limit integration tests to high-risk parts of the
larger project (either due to uncertainty of the current
implementation, or because it would be easy for later changes to
inadvertently break things). Again, my goal isn't to verify that the
implementation hasn't changed, but rather to help narrow down the
source of breakages. If the feature isn't working, the feature-level
tests should be breaking (if not, they need more detail), and then the
rest of the testing just helps you find the root cause more quickly.

HTH!
Eli



On Sun, Dec 10, 2017 at 9:14 AM, Manuel Jacob <me at manueljacob.de> wrote:
> Hi,
>
> I'm currently implementing PEP 526 [1], which requires additions to the
> interpreter from the parser down to the bytecode interpreter.  Since this is
> a common task, I want to expand the documentation a bit, adding a "how to
> extend the interpreter" recipe.
>
> One thing which I wasn't 100% sure about myself is how to test it.  Clearly
> every addition has to come with a test, preferably written before
> implementing anything.  But a few things are a bit unclear:
>
> For every change in CPython there is (hopefully) a test case in
> lib-python/*/test.  So technically there's already a test case when
> modifying the interpreter.  One reason to add app-level tests is to avoid
> re-translations.  But other than this convenience, how much app-level
> testing to we *require*?  Should we test everything up to the last detail or
> are we fine with testing the basic functionality and leaving the rest to the
> CPython test suite?
>
> A related question is which style of testing — integration-test-style or
> unit-test-style — we prefer.  Different people — also among the core
> developers — have different preferences, so we have a bit of a mix.  So in
> this concrete example, it is tempting to create a single test like
> AppTestFunctionAnnotations.test_simple from
> pypy/interpreter/test/test_syntax.py [2] and then add all the functionality
> to the interpreter until the test passes (this would be integration-style
> testing).  From the unit-style testing perspective, this test is not very
> good, because it doesn't test only the syntax, like the file name
> "test_syntax.py" would suggest.  The alternative would be to add a unit test
> for every affected part of the interpreter (parser, AST builder, AST
> validator, bytecode generation, symtable, stack effect computation, bytecode
> interpreter).
>
> What do you think?  Feel free to braindump your general thoughts (also
> outside of the syntax addition case), so I can extend other relevant parts
> of the documentation as well.  This will hopefully make it easier for new
> (but also more advanced) contributors and ensure a consistent test suite
> quality all over our code base.
>
> -Manuel
>
> [1] https://www.python.org/dev/peps/pep-0526/
> [2]
> https://bitbucket.org/pypy/pypy/src/e30b20325b579b5a92cba08d326d2e385caba2d9/pypy/interpreter/test/test_syntax.py?fileviewer=file-view-default#test_syntax.py-685
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev