[SciPy-Dev] GSoC'17 candidate - Interested in Nose to Pytest Migration.

[Karan Desai]

Hello developers,
I was out of town for a while. Came back to see there's quite a lot of
discussion on this idea. Just to acknowledge that it'll take time for me to
process all the information above. I'll give in my inputs to discussion or
contribute a patch directly in near time.
Regards,Karan.  





On Thu, Mar 2, 2017 4:48 PM, Ralf Gommers ralf.gommers at gmail.com  wrote:


On Wed, Mar 1, 2017 at 8:01 AM, Robert Kern <robert.kern at gmail.com>  wrote:
On Tue, Feb 28, 2017 at 1:31 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
> On Tue, Feb 28, 2017 at 12:08 PM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On Mon, Feb 27, 2017 at 2:32 PM, Charles R Harris <charlesr.harris at gmail.com>
wrote:
>> >
>> >
>> >
>> > On Mon, Feb 27, 2017 at 2:35 PM, Robert Kern <robert.kern at gmail.com> wrote:
>> >>
>> >> On Mon, Feb 27, 2017 at 1:03 PM, Pauli Virtanen <pav at iki.fi> wrote:
>> >> >
>> >> > Mon, 27 Feb 2017 19:13:48 +0300, Evgeni Burovski kirjoitti:
>> >> > [clip]
>> >> > > A few scattered thoughts about the project:
>> >> > >
>> >> > > - This is not exactly a transition to pytest as such. The goal is to
>> >> > > allow an alternative test runner, with minimal changes to the test
>> >> > > suite.
>> >> >
>> >> > I'd suggest we should consider actually dropping nose, if we make the
>> >> > suite pytest compatible.
>> >> >
>> >> > I think retaining nose compatibility in the numpy/scipy test suites does
>> >> > not bring advantages --- if we are planning to retain it, I would like
to
>> >> > understand why. Supporting two systems is more complicated, makes it
>> >> > harder to add new tests, and if the other one is not regularly used,
>> >> > it'll probably break often.
>> >> >
>> >> > I think we should avoid temptation to build compatibility layers, or
>> >> > custom test runners --- there's already complication in numpy.testing
>> >> > that IIRC originates from working around issues in nose, and I think not
>> >> > many remember how it works in depth.
>> >> >
>> >> > I think the needs for Scipy and Numpy are not demanding, so sticking
with
>> >> > "vanilla" pytest features and using only its native test runner would
>> >> > sound best. The main requirement is numerical assert functions, but
AFAIK
>> >> > the ones in numpy.testing are pytest compatible (and independent of the
>> >> > nose machinery).
>> >>
>> >> If we're migrating test runners, I'd rather drop all dependencies and
target `python -m unittest discover` as the lowest common denominator rather
than target a fuzzy "vanilla" subset of pytest in particular.
>> >
>> > I'd certainly expect to make full use of the pytest features, why use it
otherwise? There is a reason that both nose and pytest extended unittest. The
only advantage I see to unittest is that it is less likely to become abandon
ware.
>>
>> Well, that's definitely not what Pauli was advocating, and my response was
intending to clarify his position.
>>
>> >> This may well be the technical effect of what you are describing, but I
think it's worth explicitly stating that focus.
>> >
>> > I don't think that us where we were headed. Can you make the case for
unittest?
>>
>> There are two places where nose and pytest provide features; they are each a
*test runner* and also a *test framework*. As a test runner, they provide
features for discovering tests (essentially, they let you easily point to a
subset of tests to run) and reporting the test results in a variety of
user-friendly ways. Test discovery functionality works with plain-old-TestCases,
and pretty much any test runner works with the plain-old-TestCases. These
features face the user who is *running* the tests.
>>
>> The other place they provide features is as test frameworks. This is where
things like the generator tests and additional fixtures come in (i.e.
setup_class, setup_module, etc.). These require you to write your tests in
framework-specific ways. Those written for nose don't necessarily work with
those written for pytest and vice versa. And any other plain test runner is
right out. These features face the developer who is *writing* the tests.
>>
>> Test discovery was the main reason we started using nose over plain unittest.
At that time (and incidentally, when nose and pytest began), `python -m unittest
discover` didn't exist.
>
> As Josef said, "import scipy; scipy.test()" must work (always useful, but
especially important on Windows).

Okay. And?
No and, just pointing out that `python -m unittest discover` doesn't cut it.


>> To run unit tests, you had to manually collect the TestCases into a TestSuite
and write your own logic for configuring any subsets from user input. It was a
pain in the ass for large hierarchical packages like numpy and scipy. The test
framework features of nose like generator tests were nice bonuses, but were not
the main motivating factor. [Source: me; it was Jarrod and I who made the
decision to use nose at a numpy sprint in Berkeley.] 
>>
>> If we're going to make a switch, I'd rather give up those framework features
(we don't use all that many) in order to get agnosticism on the test runner
front. Being agnostic to the test runner lets us optimize for multiple use cases
simultaneously. That is, the best test runner for "run the whole suite and
record every detail with coverage under time constraints under Travis CI" is
often different from what a developer wants for "run just the one test module
for the thing I'm working on and report to me only what breaks as fast as
possible with minimal clutter". 
>
> I don't believe we can live with just plain unittest, we want to move to a
better maintained framework and gain some features in this GSoC project, not go
backwards in functionality.

Ah, okay. The way the project abstract is written, this is not clear. It reads
as if you want pytest as a mere additional possible test runner alongside nose.
Instead, it seems that you want to migrate away from nose and to pytest,
wholesale. I'm fine with that. But get Pauli on board with going beyond
"vanilla" pytest and rewrite the abstract. Drop the bit about ideally running
the test suite with both nose and pytest. 
Fair point. Will try to clean up the abstract over the weekend.

If you're looking to gain functionality, there's no point, just maintenance
headaches. If you want some runner-agnosticism, then unittest is the layer to do
it on. Maintaining runner-agnosticism for just two runners (only one of which is
maintained) is a wasted opportunity. That's my main contention.

You're right. Runner agnosticism is not what we were aiming at, but if we were
then it'd probably have to be plain unittest.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20170302/b3811235/attachment.html>