[SciPy-Dev] GSoC'17 candidate - Interested in Nose to Pytest Migration.

[Ralf Gommers]

On Tue, Feb 28, 2017 at 12:08 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Mon, Feb 27, 2017 at 2:32 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
> >
> >
> >
> > On Mon, Feb 27, 2017 at 2:35 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Mon, Feb 27, 2017 at 1:03 PM, Pauli Virtanen <pav at iki.fi> wrote:
> >> >
> >> > Mon, 27 Feb 2017 19:13:48 +0300, Evgeni Burovski kirjoitti:
> >> > [clip]
> >> > > A few scattered thoughts about the project:
> >> > >
> >> > > - This is not exactly a transition to pytest as such. The goal is to
> >> > > allow an alternative test runner, with minimal changes to the test
> >> > > suite.
> >> >
> >> > I'd suggest we should consider actually dropping nose, if we make the
> >> > suite pytest compatible.
> >> >
> >> > I think retaining nose compatibility in the numpy/scipy test suites
> does
> >> > not bring advantages --- if we are planning to retain it, I would
> like to
> >> > understand why. Supporting two systems is more complicated, makes it
> >> > harder to add new tests, and if the other one is not regularly used,
> >> > it'll probably break often.
> >> >
> >> > I think we should avoid temptation to build compatibility layers, or
> >> > custom test runners --- there's already complication in numpy.testing
> >> > that IIRC originates from working around issues in nose, and I think
> not
> >> > many remember how it works in depth.
> >> >
> >> > I think the needs for Scipy and Numpy are not demanding, so sticking
> with
> >> > "vanilla" pytest features and using only its native test runner would
> >> > sound best. The main requirement is numerical assert functions, but
> AFAIK
> >> > the ones in numpy.testing are pytest compatible (and independent of
> the
> >> > nose machinery).
> >>
> >> If we're migrating test runners, I'd rather drop all dependencies and
> target `python -m unittest discover` as the lowest common denominator
> rather than target a fuzzy "vanilla" subset of pytest in particular.
> >
> > I'd certainly expect to make full use of the pytest features, why use it
> otherwise? There is a reason that both nose and pytest extended unittest.
> The only advantage I see to unittest is that it is less likely to become
> abandon ware.
>
> Well, that's definitely not what Pauli was advocating, and my response was
> intending to clarify his position.
>
> >> This may well be the technical effect of what you are describing, but I
> think it's worth explicitly stating that focus.
> >
> > I don't think that us where we were headed. Can you make the case for
> unittest?
>
> There are two places where nose and pytest provide features; they are each
> a *test runner* and also a *test framework*. As a test runner, they provide
> features for discovering tests (essentially, they let you easily point to a
> subset of tests to run) and reporting the test results in a variety of
> user-friendly ways. Test discovery functionality works with
> plain-old-TestCases, and pretty much any test runner works with the
> plain-old-TestCases. These features face the user who is *running* the
> tests.
>
> The other place they provide features is as test frameworks. This is where
> things like the generator tests and additional fixtures come in (i.e.
> setup_class, setup_module, etc.). These require you to write your tests in
> framework-specific ways. Those written for nose don't necessarily work with
> those written for pytest and vice versa. And any other plain test runner is
> right out. These features face the developer who is *writing* the tests.
>
> Test discovery was the main reason we started using nose over plain
> unittest. At that time (and incidentally, when nose and pytest began),
> `python -m unittest discover` didn't exist.
>

As Josef said, "import scipy; scipy.test()" must work (always useful, but
especially important on Windows).


> To run unit tests, you had to manually collect the TestCases into a
> TestSuite and write your own logic for configuring any subsets from user
> input. It was a pain in the ass for large hierarchical packages like numpy
> and scipy. The test framework features of nose like generator tests were
> nice bonuses, but were not the main motivating factor. [Source: me; it was
> Jarrod and I who made the decision to use nose at a numpy sprint in
> Berkeley.]
>
> If we're going to make a switch, I'd rather give up those framework
> features (we don't use all that many) in order to get agnosticism on the
> test runner front. Being agnostic to the test runner lets us optimize for
> multiple use cases simultaneously. That is, the best test runner for "run
> the whole suite and record every detail with coverage under time
> constraints under Travis CI" is often different from what a developer wants
> for "run just the one test module for the thing I'm working on and report
> to me only what breaks as fast as possible with minimal clutter".
>

I don't believe we can live with just plain unittest, we want to move to a
better maintained framework and gain some features in this GSoC project,
not go backwards in functionality.

Some other issues:
- unittest itself isn't going to gain any form of parametric testing any
time soon (https://bugs.python.org/issue7897) and we use those extensively
- we need to keep test functions (i.e. not methods of a TestCase subclass)
working
- we need something like numpy.testing.NoseTester to get us the right
behavior for warnings, print versions/paths, etc.
- there's decorators like assert_raises_regex, setastest, etc. that we want
- we want to gain a test timer: https://github.com/scipy/scipy/issues/6989

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170228/725cf7b4/attachment.html>