[SciPy-Dev] GSoC'17 candidate - Interested in Nose to Pytest Migration.

[Ralf Gommers]

On Wed, Mar 1, 2017 at 8:01 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On Tue, Feb 28, 2017 at 1:31 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> > On Tue, Feb 28, 2017 at 12:08 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >>
> >> On Mon, Feb 27, 2017 at 2:32 PM, Charles R Harris <
> charlesr.harris at gmail.com> wrote:
> >> >
> >> >
> >> >
> >> > On Mon, Feb 27, 2017 at 2:35 PM, Robert Kern <robert.kern at gmail.com>
> wrote:
> >> >>
> >> >> On Mon, Feb 27, 2017 at 1:03 PM, Pauli Virtanen <pav at iki.fi> wrote:
> >> >> >
> >> >> > Mon, 27 Feb 2017 19:13:48 +0300, Evgeni Burovski kirjoitti:
> >> >> > [clip]
> >> >> > > A few scattered thoughts about the project:
> >> >> > >
> >> >> > > - This is not exactly a transition to pytest as such. The goal
> is to
> >> >> > > allow an alternative test runner, with minimal changes to the
> test
> >> >> > > suite.
> >> >> >
> >> >> > I'd suggest we should consider actually dropping nose, if we make
> the
> >> >> > suite pytest compatible.
> >> >> >
> >> >> > I think retaining nose compatibility in the numpy/scipy test
> suites does
> >> >> > not bring advantages --- if we are planning to retain it, I would
> like to
> >> >> > understand why. Supporting two systems is more complicated, makes
> it
> >> >> > harder to add new tests, and if the other one is not regularly
> used,
> >> >> > it'll probably break often.
> >> >> >
> >> >> > I think we should avoid temptation to build compatibility layers,
> or
> >> >> > custom test runners --- there's already complication in
> numpy.testing
> >> >> > that IIRC originates from working around issues in nose, and I
> think not
> >> >> > many remember how it works in depth.
> >> >> >
> >> >> > I think the needs for Scipy and Numpy are not demanding, so
> sticking with
> >> >> > "vanilla" pytest features and using only its native test runner
> would
> >> >> > sound best. The main requirement is numerical assert functions,
> but AFAIK
> >> >> > the ones in numpy.testing are pytest compatible (and independent
> of the
> >> >> > nose machinery).
> >> >>
> >> >> If we're migrating test runners, I'd rather drop all dependencies
> and target `python -m unittest discover` as the lowest common denominator
> rather than target a fuzzy "vanilla" subset of pytest in particular.
> >> >
> >> > I'd certainly expect to make full use of the pytest features, why use
> it otherwise? There is a reason that both nose and pytest extended
> unittest. The only advantage I see to unittest is that it is less likely to
> become abandon ware.
> >>
> >> Well, that's definitely not what Pauli was advocating, and my response
> was intending to clarify his position.
> >>
> >> >> This may well be the technical effect of what you are describing,
> but I think it's worth explicitly stating that focus.
> >> >
> >> > I don't think that us where we were headed. Can you make the case for
> unittest?
> >>
> >> There are two places where nose and pytest provide features; they are
> each a *test runner* and also a *test framework*. As a test runner, they
> provide features for discovering tests (essentially, they let you easily
> point to a subset of tests to run) and reporting the test results in a
> variety of user-friendly ways. Test discovery functionality works with
> plain-old-TestCases, and pretty much any test runner works with the
> plain-old-TestCases. These features face the user who is *running* the
> tests.
> >>
> >> The other place they provide features is as test frameworks. This is
> where things like the generator tests and additional fixtures come in (i.e.
> setup_class, setup_module, etc.). These require you to write your tests in
> framework-specific ways. Those written for nose don't necessarily work with
> those written for pytest and vice versa. And any other plain test runner is
> right out. These features face the developer who is *writing* the tests.
> >>
> >> Test discovery was the main reason we started using nose over plain
> unittest. At that time (and incidentally, when nose and pytest began),
> `python -m unittest discover` didn't exist.
> >
> > As Josef said, "import scipy; scipy.test()" must work (always useful,
> but especially important on Windows).
>
> Okay. And?
>

No and, just pointing out that `python -m unittest discover` doesn't cut it.


> >> To run unit tests, you had to manually collect the TestCases into a
> TestSuite and write your own logic for configuring any subsets from user
> input. It was a pain in the ass for large hierarchical packages like numpy
> and scipy. The test framework features of nose like generator tests were
> nice bonuses, but were not the main motivating factor. [Source: me; it was
> Jarrod and I who made the decision to use nose at a numpy sprint in
> Berkeley.]
> >>
> >> If we're going to make a switch, I'd rather give up those framework
> features (we don't use all that many) in order to get agnosticism on the
> test runner front. Being agnostic to the test runner lets us optimize for
> multiple use cases simultaneously. That is, the best test runner for "run
> the whole suite and record every detail with coverage under time
> constraints under Travis CI" is often different from what a developer wants
> for "run just the one test module for the thing I'm working on and report
> to me only what breaks as fast as possible with minimal clutter".
> >
> > I don't believe we can live with just plain unittest, we want to move to
> a better maintained framework and gain some features in this GSoC project,
> not go backwards in functionality.
>
> Ah, okay. The way the project abstract is written, this is not clear. It
> reads as if you want pytest as a mere additional possible test runner
> alongside nose. Instead, it seems that you want to migrate away from nose
> and to pytest, wholesale. I'm fine with that. But get Pauli on board with
> going beyond "vanilla" pytest and rewrite the abstract. Drop the bit about
> ideally running the test suite with both nose and pytest.
>

Fair point. Will try to clean up the abstract over the weekend.


> If you're looking to gain functionality, there's no point, just
> maintenance headaches. If you want some runner-agnosticism, then unittest
> is the layer to do it on. Maintaining runner-agnosticism for just two
> runners (only one of which is maintained) is a wasted opportunity. That's
> my main contention.
>

You're right. Runner agnosticism is not what we were aiming at, but if we
were then it'd probably have to be plain unittest.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170303/f398f140/attachment.html>