[pytest-dev] Flaky tests

[Eduardo Schettino]

Hi,

I will just throw out another idea that I had about handling flaky tests.

I think it would be nice to mark a test as "flaky" and than treat it
differently if it fails.

For example, once a test is marked as flaky you could choose to ignore
flaky test failures and maybe report them
as an "expect failure" or some other custom status. Or still consider it as
failure if run on a "strict mode".

regards,
  Eduardo

On Sun, Aug 9, 2015 at 8:54 AM, Anatoly Bubenkov <bubenkoff at gmail.com>
wrote:

> Hi all,
> flaky tests are often really hard to make not flaky, so I've got the idea
> to just retry failed tests several times automatically and only then report
> the failure.
> I was not the only one of course: there's nice initiative in face of
> https://github.com/box/flaky
> The big downside of that plugin's approach, is that only 'call' phase is
> retried, while 'setup' is not re-executed and so all function-scoped
> fixtures stay as is from previous try.
> This is not acceptable if you use fixtures intensively of course.
> I created an issue where I have a snippet how it could work
> https://github.com/box/flaky/issues/53
> But then realized that it doesn't work properly still: when it's the only
> test collected which we're going to retry, all fixtures, including
> session-scoped are being resetup as well!
> Then looked more and couldn't find any good solutions to actually very
> simple question: how do i rerun the test in the same test run?
> Any help is appreciated. I think making that plugin work or even making it
> available in the core would be a great improvement.
> Thanks!
>
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20150809/d5a076b7/attachment.html>