[SciPy-Dev] GSoC'17 candidate - Interested in Nose to Pytest Migration.

[Robert Kern]

On Tue, Feb 28, 2017 at 1:31 AM, Ralf Gommers <ralf.gommers at gmail.com>
wrote:
>
> On Tue, Feb 28, 2017 at 12:08 PM, Robert Kern <robert.kern at gmail.com>
wrote:
>>
>> On Mon, Feb 27, 2017 at 2:32 PM, Charles R Harris <
charlesr.harris at gmail.com> wrote:
>> >
>> >
>> >
>> > On Mon, Feb 27, 2017 at 2:35 PM, Robert Kern <robert.kern at gmail.com>
wrote:
>> >>
>> >> On Mon, Feb 27, 2017 at 1:03 PM, Pauli Virtanen <pav at iki.fi> wrote:
>> >> >
>> >> > Mon, 27 Feb 2017 19:13:48 +0300, Evgeni Burovski kirjoitti:
>> >> > [clip]
>> >> > > A few scattered thoughts about the project:
>> >> > >
>> >> > > - This is not exactly a transition to pytest as such. The goal is
to
>> >> > > allow an alternative test runner, with minimal changes to the test
>> >> > > suite.
>> >> >
>> >> > I'd suggest we should consider actually dropping nose, if we make
the
>> >> > suite pytest compatible.
>> >> >
>> >> > I think retaining nose compatibility in the numpy/scipy test suites
does
>> >> > not bring advantages --- if we are planning to retain it, I would
like to
>> >> > understand why. Supporting two systems is more complicated, makes it
>> >> > harder to add new tests, and if the other one is not regularly used,
>> >> > it'll probably break often.
>> >> >
>> >> > I think we should avoid temptation to build compatibility layers, or
>> >> > custom test runners --- there's already complication in
numpy.testing
>> >> > that IIRC originates from working around issues in nose, and I
think not
>> >> > many remember how it works in depth.
>> >> >
>> >> > I think the needs for Scipy and Numpy are not demanding, so
sticking with
>> >> > "vanilla" pytest features and using only its native test runner
would
>> >> > sound best. The main requirement is numerical assert functions, but
AFAIK
>> >> > the ones in numpy.testing are pytest compatible (and independent of
the
>> >> > nose machinery).
>> >>
>> >> If we're migrating test runners, I'd rather drop all dependencies and
target `python -m unittest discover` as the lowest common denominator
rather than target a fuzzy "vanilla" subset of pytest in particular.
>> >
>> > I'd certainly expect to make full use of the pytest features, why use
it otherwise? There is a reason that both nose and pytest extended
unittest. The only advantage I see to unittest is that it is less likely to
become abandon ware.
>>
>> Well, that's definitely not what Pauli was advocating, and my response
was intending to clarify his position.
>>
>> >> This may well be the technical effect of what you are describing, but
I think it's worth explicitly stating that focus.
>> >
>> > I don't think that us where we were headed. Can you make the case for
unittest?
>>
>> There are two places where nose and pytest provide features; they are
each a *test runner* and also a *test framework*. As a test runner, they
provide features for discovering tests (essentially, they let you easily
point to a subset of tests to run) and reporting the test results in a
variety of user-friendly ways. Test discovery functionality works with
plain-old-TestCases, and pretty much any test runner works with the
plain-old-TestCases. These features face the user who is *running* the
tests.
>>
>> The other place they provide features is as test frameworks. This is
where things like the generator tests and additional fixtures come in (i.e.
setup_class, setup_module, etc.). These require you to write your tests in
framework-specific ways. Those written for nose don't necessarily work with
those written for pytest and vice versa. And any other plain test runner is
right out. These features face the developer who is *writing* the tests.
>>
>> Test discovery was the main reason we started using nose over plain
unittest. At that time (and incidentally, when nose and pytest began),
`python -m unittest discover` didn't exist.
>
> As Josef said, "import scipy; scipy.test()" must work (always useful, but
especially important on Windows).

Okay. And?

>> To run unit tests, you had to manually collect the TestCases into a
TestSuite and write your own logic for configuring any subsets from user
input. It was a pain in the ass for large hierarchical packages like numpy
and scipy. The test framework features of nose like generator tests were
nice bonuses, but were not the main motivating factor. [Source: me; it was
Jarrod and I who made the decision to use nose at a numpy sprint in
Berkeley.]
>>
>> If we're going to make a switch, I'd rather give up those framework
features (we don't use all that many) in order to get agnosticism on the
test runner front. Being agnostic to the test runner lets us optimize for
multiple use cases simultaneously. That is, the best test runner for "run
the whole suite and record every detail with coverage under time
constraints under Travis CI" is often different from what a developer wants
for "run just the one test module for the thing I'm working on and report
to me only what breaks as fast as possible with minimal clutter".
>
> I don't believe we can live with just plain unittest, we want to move to
a better maintained framework and gain some features in this GSoC project,
not go backwards in functionality.

Ah, okay. The way the project abstract is written, this is not clear. It
reads as if you want pytest as a mere additional possible test runner
alongside nose. Instead, it seems that you want to migrate away from nose
and to pytest, wholesale. I'm fine with that. But get Pauli on board with
going beyond "vanilla" pytest and rewrite the abstract. Drop the bit about
ideally running the test suite with both nose and pytest. If you're looking
to gain functionality, there's no point, just maintenance headaches. If you
want some runner-agnosticism, then unittest is the layer to do it on.
Maintaining runner-agnosticism for just two runners (only one of which is
maintained) is a wasted opportunity. That's my main contention.

> Some other issues:
> - unittest itself isn't going to gain any form of parametric testing any
time soon (https://bugs.python.org/issue7897) and we use those extensively
> - we need to keep test functions (i.e. not methods of a TestCase
subclass) working
> - we need something like numpy.testing.NoseTester to get us the right
behavior for warnings, print versions/paths, etc.

About the same work to accomplish with unittest as pytest. Despite the
name, there isn't much nose-or-fancy-framework-specific in there.

> - there's decorators like assert_raises_regex, setastest, etc. that we
want

Just for the record, things like assert_raises_regex are
runner/framework-agnostic, and setastest doesn't matter if you are using
TestCases. ;-)

> - we want to gain a test timer: https://github.com/scipy/scipy/issues/6989

Also for the record, this is just a runner feature and has no real bearing
on how the tests are written. A suite of plain-old-TestCases can be run by
`pytest --durations` just fine.

<sigh> I'm gonna miss nose-progressive.
  https://pypi.python.org/pypi/nose-progressive/

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170228/63f11dd5/attachment.html>