[pytest-dev] document outlining added values

[Craig de Stigter]

Wow, I sure wish I'd found that earlier. I swear I'd been looking for this
for years before I started writing my own :) That said, I've learned an
awful lot by writing my own; I wouldn't have been able to do the GDAL
conversion without writing that first.

Sivan, I'm sorry for misinterpreting your original post and (inadvertently)
derailing your thread :) I'll try to answer the original question.

We converted our unittest/django test suite to pytest style over the last
few years (recently very quickly as a result of this conversion script).
Here's a list of advantages we've seen. Some of these are to do with the
pytest *runner* and some are to do with the particular style of pytest
tests themselves.

Syntactical advantages:

   - assertion syntax is more pythonic (less camelCase)
   - assertion syntax is less verbose, and equally if not more readable
   - assertion *messages* are rarely required with pytest, because it
   prints the sub-expressions of the expressions used in the assert statement.

Function-based test advantages:

   - With a large enough test suite, each test case class becomes a woolly
   maze of multiple inheritance with mixins. We found this more confusing than
   the explicit dependency injection that pytest's fixtures offer
   - Having all functions in TestCase class objects makes it annoying to
   run a *particular* test; you have to type the classname as well as the
   module and test function name

Performance:

   - Not *exactly* related but: Django's XML/JSON fixture files are
   terribly slow to load (or at least, were when I last checked, probably
   about django 1.8). XML was better than JSON but neither approach the speed
   you can get by creating objects manually via the ORM. Moving to pytest was
   a good opportunity to move these fixture files into pytest fixtures using
   the Django ORM. (Note: there are other ways to avoid this - you could use
   factory_boy or something and avoid django fixture files)

Customisation:

   - We've found it much easier to extend the functionality of our test
   suite. For example, recently we added a new version to our API (all
   `/api/v1/endpoint/` things can be accessed via `/api/v2/endpoint/` too
   now). We trivially parametrized our test-client fixture, to run all API
   tests against both endpoints, and transparently rewrite the version numbers
   in the URLs. This wouldn't have been easily possible with a standard django
   test suite.
   - Pytest makes it easy to run fixtures with different scopes, whereas
   unittest-based testing assumes all setup happens for a particular class or
   function. In particular, session-scope fixtures allow us to do setup once
   for the whole test run.
   - It's easy to setup things *by default*. For example, we wanted to
   *remove* access to our storage volume during tests by default, so that
   we could verify that nothing touched it that didn't *need* to. This was
   easy with pytest - we just created an autouse fixture that runs before each
   test and ensures the storage volume is unavailable. The fixture is removed
   during `pytest_collection_modifyitems` if a particular marker is present to
   say a test needs storage access.
   - It's easy to customise what tests will be run, either with custom
   command-line flags, environment variables, or whatever you want. Just add a
   pytest_collection_modifyitems hook and tweak the list. You can skip tests,
   reorder tests, add fixtures, add markers... whatever you want. I don't know
   if this is even possible with the django test runner without heavy patching.


I hope that's helpful. Switching to pytest has been a real breath of fresh
air and has greatly accelerated our *desire* to write tests, which I think
is important. It makes testing less of a drag, and maybe even fun :)

Craig de Stigter
Platform Engineer @ https://koordinates.com


On Wed, 5 Dec 2018 at 01:12 Florian Bruhin <me at the-compiler.org> wrote:

> On Tue, Dec 04, 2018 at 12:54:50AM +0000, Craig de Stigter wrote:
> > > gradually porting all tests
> >
> > Mostly-automated conversion is possible, so it doesn't have to be a
> gradual
> > process.
> >
> > I wrote a script which converts xunit-style assertions into pytest ones:
> > https://github.com/craigds/decrapify/blob/master/pytestify.py
>
> Yet another one? ;-)
>
> https://github.com/pytest-dev/unittest2pytest
> https://github.com/dropbox/dbx-unittest2pytest
>
> Florian
>
> --
> https://www.qutebrowser.org | me at the-compiler.org (Mail/XMPP)
>    GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc
>          I love long mails! | https://email.is-not-s.ms/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20181205/a08c85a4/attachment.html>