[SciPy-Dev] GSoC'17 candidate - Interested in Nose to Pytest Migration.

[Matthew Brett]

Hi,

On Tue, Feb 28, 2017 at 11:01 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On Tue, Feb 28, 2017 at 1:31 AM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>>
>> On Tue, Feb 28, 2017 at 12:08 PM, Robert Kern <robert.kern at gmail.com>
>> wrote:
>>>
>>> On Mon, Feb 27, 2017 at 2:32 PM, Charles R Harris
>>> <charlesr.harris at gmail.com> wrote:
>>> >
>>> >
>>> >
>>> > On Mon, Feb 27, 2017 at 2:35 PM, Robert Kern <robert.kern at gmail.com>
>>> > wrote:
>>> >>
>>> >> On Mon, Feb 27, 2017 at 1:03 PM, Pauli Virtanen <pav at iki.fi> wrote:
>>> >> >
>>> >> > Mon, 27 Feb 2017 19:13:48 +0300, Evgeni Burovski kirjoitti:
>>> >> > [clip]
>>> >> > > A few scattered thoughts about the project:
>>> >> > >
>>> >> > > - This is not exactly a transition to pytest as such. The goal is
>>> >> > > to
>>> >> > > allow an alternative test runner, with minimal changes to the test
>>> >> > > suite.
>>> >> >
>>> >> > I'd suggest we should consider actually dropping nose, if we make
>>> >> > the
>>> >> > suite pytest compatible.
>>> >> >
>>> >> > I think retaining nose compatibility in the numpy/scipy test suites
>>> >> > does
>>> >> > not bring advantages --- if we are planning to retain it, I would
>>> >> > like to
>>> >> > understand why. Supporting two systems is more complicated, makes it
>>> >> > harder to add new tests, and if the other one is not regularly used,
>>> >> > it'll probably break often.
>>> >> >
>>> >> > I think we should avoid temptation to build compatibility layers, or
>>> >> > custom test runners --- there's already complication in
>>> >> > numpy.testing
>>> >> > that IIRC originates from working around issues in nose, and I think
>>> >> > not
>>> >> > many remember how it works in depth.
>>> >> >
>>> >> > I think the needs for Scipy and Numpy are not demanding, so sticking
>>> >> > with
>>> >> > "vanilla" pytest features and using only its native test runner
>>> >> > would
>>> >> > sound best. The main requirement is numerical assert functions, but
>>> >> > AFAIK
>>> >> > the ones in numpy.testing are pytest compatible (and independent of
>>> >> > the
>>> >> > nose machinery).
>>> >>
>>> >> If we're migrating test runners, I'd rather drop all dependencies and
>>> >> target `python -m unittest discover` as the lowest common denominator rather
>>> >> than target a fuzzy "vanilla" subset of pytest in particular.
>>> >
>>> > I'd certainly expect to make full use of the pytest features, why use
>>> > it otherwise? There is a reason that both nose and pytest extended unittest.
>>> > The only advantage I see to unittest is that it is less likely to become
>>> > abandon ware.
>>>
>>> Well, that's definitely not what Pauli was advocating, and my response
>>> was intending to clarify his position.
>>>
>>> >> This may well be the technical effect of what you are describing, but
>>> >> I think it's worth explicitly stating that focus.
>>> >
>>> > I don't think that us where we were headed. Can you make the case for
>>> > unittest?
>>>
>>> There are two places where nose and pytest provide features; they are
>>> each a *test runner* and also a *test framework*. As a test runner, they
>>> provide features for discovering tests (essentially, they let you easily
>>> point to a subset of tests to run) and reporting the test results in a
>>> variety of user-friendly ways. Test discovery functionality works with
>>> plain-old-TestCases, and pretty much any test runner works with the
>>> plain-old-TestCases. These features face the user who is *running* the
>>> tests.
>>>
>>> The other place they provide features is as test frameworks. This is
>>> where things like the generator tests and additional fixtures come in (i.e.
>>> setup_class, setup_module, etc.). These require you to write your tests in
>>> framework-specific ways. Those written for nose don't necessarily work with
>>> those written for pytest and vice versa. And any other plain test runner is
>>> right out. These features face the developer who is *writing* the tests.
>>>
>>> Test discovery was the main reason we started using nose over plain
>>> unittest. At that time (and incidentally, when nose and pytest began),
>>> `python -m unittest discover` didn't exist.
>>
>> As Josef said, "import scipy; scipy.test()" must work (always useful, but
>> especially important on Windows).
>
> Okay. And?
>
>>> To run unit tests, you had to manually collect the TestCases into a
>>> TestSuite and write your own logic for configuring any subsets from user
>>> input. It was a pain in the ass for large hierarchical packages like numpy
>>> and scipy. The test framework features of nose like generator tests were
>>> nice bonuses, but were not the main motivating factor. [Source: me; it was
>>> Jarrod and I who made the decision to use nose at a numpy sprint in
>>> Berkeley.]
>>>
>>> If we're going to make a switch, I'd rather give up those framework
>>> features (we don't use all that many) in order to get agnosticism on the
>>> test runner front. Being agnostic to the test runner lets us optimize for
>>> multiple use cases simultaneously. That is, the best test runner for "run
>>> the whole suite and record every detail with coverage under time constraints
>>> under Travis CI" is often different from what a developer wants for "run
>>> just the one test module for the thing I'm working on and report to me only
>>> what breaks as fast as possible with minimal clutter".
>>
>> I don't believe we can live with just plain unittest, we want to move to a
>> better maintained framework and gain some features in this GSoC project, not
>> go backwards in functionality.
>
> Ah, okay. The way the project abstract is written, this is not clear. It
> reads as if you want pytest as a mere additional possible test runner
> alongside nose. Instead, it seems that you want to migrate away from nose
> and to pytest, wholesale. I'm fine with that. But get Pauli on board with
> going beyond "vanilla" pytest and rewrite the abstract. Drop the bit about
> ideally running the test suite with both nose and pytest. If you're looking
> to gain functionality, there's no point, just maintenance headaches. If you
> want some runner-agnosticism, then unittest is the layer to do it on.
> Maintaining runner-agnosticism for just two runners (only one of which is
> maintained) is a wasted opportunity. That's my main contention.

That seems reasonable.   Maybe we have a problem here, in that we
might prefer a big-bang switch from nose to pytest, but the GSoC
format has a strong requirement for gradual progress over many weeks.
That may mean some serious drag from keeping a long-lived parallel
development branch up to date.  At least it will require some thought.

Matplotlib and scikit-image just migrated to pytest.  I've Cc'ed
Thomas Caswell and Stefan van der Walt, hoping they have some
experience to share about the migration.

See you,

Matthew