[pytest-dev] My gripes with pytest and tox

[Ronny Pfannschmidt]

Hi Florian,

below i threw in some thoughts and strong opinions,
take with a grain of salt and feel free to tear them down :)

best, Ronny

Am 10.06.2016 um 09:12 schrieb Florian Bruhin:
> Hi,
>
> What's following here are a few thoughts about things I miss or
> dislike about pytest, plugins I use, and tox. This is a list of things
> I personally would like to get fixed during the sprint, but of course
> that's not the only thing I want to work on, then I wouldn't need a
> sprint with other people. ;)
>
> Let me mention again this is my *personal* view of things. You might
> disagree with some of it. That's okay. Let's not turn this into a
> bike-shedding for now, though feel free to reply privately if needed.
>
> This also is not a rant. It's just a reminder for myself and a todo
> list. All of this software is awesome! Those are all minor annoyances.
>
> Some of those items have issues open, some of them I haven't told
> anyone about yet. A few of them might not be solvable at all.
>
> I thought for some while whether to make this public or not, but
> figured, hey, why not.
>
> In no particular order:
>
> pytest
>     - doesn't have an easy way to do printf-style debugging
>       (without -s)
thats per design - you are supposed to leave prints in and use the
output capture to get context
i would prefer to extend on that instead of adding a new more limited
way just because its convenient in some cases
>     - prints the exception before all other report output instead of
>       after
perhaps it makes sense to order things based on number of lines, have a
way to take control of that
different kinds of testsuites have different kinds of output and debug help

for somethings the exceptions are a better help, for others output
traces are
>     - segfaults in my testsuite when using pdb (haven't investigated
>       yet, pdb++ works)
i suspect a potential readline issue (pdb++ uses pyrepl instead of
readline) please report
>     - blows up in weird ways with ImportErrors in plugins
thats indeed a pluggy gripe, IMHO we should put pluggy under pytest-dev
and extend it (i#d also like to cythonize it and publish
manylinux/windows wheels)

>     - too much boilerplate with parametrized fixtures
>     - too much boilerplate with parametrized tests (what about keyword
>       arguments?)
examples for those 2 items please, i saw them as convenient and
boilerplate-removing and i'd love to see your new uses and pain points
>     - has a weirdly structured documentation
agreed :/
>     - is inconsistent with types and attributes in all of its API
thats a product of growing more and more features while staying backward
compatible, i'd love to collect, revisit and clean that up as well
>     - has an unintuitive warning system, for devs and users
i think i have been more than vocal about being really unhappy about the
warning system
>     - doesn't work on Windows with colors (for me), prints black on
>       black
afair that needs colorama what can we do to improve?
>     - hides python warnings
i consider that a bad bug and gripe
personally i'd like to integrate python warnings and pytest warnings
under a consistent track-able and python warnings compatible api
>
> pytest-qt
>     - doesn't have tests yet for its modeltest branch, which is why it
>       isn't merged yet
>     - doesn't have a way to check arguments of signals
>     - doesn't work with PySide2 (which is in development)
>
> pytest-bdd
>     - doesn't have an easy way to setup/teardown things before/after
>       all tests in a .feature file
>     - doesn't have a way of parametrizing tests I can get in my head
>     - doesn't support markers with arguments in .feature files
>     - doesn't have gherkin reporting
>
>     thinking about writing an alternative which integrates an existing
>     python bdd framework (behave or lettuce) to profit from everything
>     those offer
>
> pytest-catchlog
>     - is still not in pytest-dev or core or renamed to -logging
>     - still doesn't have pytest-logging functionality integrated
>     - has no easy way to check a list of log messages
>     - has no way to fail tests on unexpected error logging messages
>
> pytest-xdist
>     - breaks my testsuite in ways I don't understand
>     - doesn't seem to have an easy way for per-job fixtures
xdist has no "jobs" - as of now its simply running multiple sessions and
schedules items between them based on strategies,
my basic opinion is that the codebase completely outgrew the initial
design and needs major internal changes
and those a re pretty expensive in terms of time and its also not clear
what model to use

i'd love to discuss at the sprint
>
> pytest-xvfb
>     - is broken with Qt4
>
> plugin ideas
>     - good profiling of tests:
>       What's slow in my testsuite?
>       What's slow in my application?
>     - new BDD plugin maybe, see above
>
> tox
>     - takes too long until my tests start
>     - doesn't detect changes in requirements.txt files
>     - doesn't support comments in deps= with newer pip's
>     - is too verbose
>       (too much noise with single-line output, needs -q)
>     - is too quiet (with -v I want to see pip output)
>
> Again - no offense to anyone, and I don't intend to start a
> bike-shedding ;)
>
> Looking forward to the sprint!
>
> Florian
>
>
>
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20160610/7fcf6436/attachment.html>