[pytest-dev] Flaky tests

[Anatoly Bubenkov]

Hi all,
flaky tests are often really hard to make not flaky, so I've got the idea
to just retry failed tests several times automatically and only then report
the failure.
I was not the only one of course: there's nice initiative in face of
https://github.com/box/flaky
The big downside of that plugin's approach, is that only 'call' phase is
retried, while 'setup' is not re-executed and so all function-scoped
fixtures stay as is from previous try.
This is not acceptable if you use fixtures intensively of course.
I created an issue where I have a snippet how it could work
https://github.com/box/flaky/issues/53
But then realized that it doesn't work properly still: when it's the only
test collected which we're going to retry, all fixtures, including
session-scoped are being resetup as well!
Then looked more and couldn't find any good solutions to actually very
simple question: how do i rerun the test in the same test run?
Any help is appreciated. I think making that plugin work or even making it
available in the core would be a great improvement.
Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20150809/a8ae15fc/attachment.html>