[pytest-dev] xdist grouping: request for user stories

[Bruno Oliveira]

Hi all,

On Thu, Oct 23, 2014 at 11:48 AM, holger krekel <holger at merlinux.eu> wrote:
> Bruno Oliveira did a PR that satisfies his company's particular
requirements, in that GUI tests must run after all others and pinned to a
particular note.

To describe my requirements in a little more detail:

We have a lot of GUI tests and most (say, 95%) work in parallel without any
problems.

A few of them though deal with window input focus, and those tests
specifically can't run in parallel with other tests. For example, a test
might:

1. create a widget and display it
2. edit a gui control, obtaining input focus
3. lose edit focus on purpose during the test, asserting that some other
behavior occurs in response to that

When this type of test runs in parallel with others, sometimes the other
tests will pop up their own widgets and the first test will lose input
focus at an unexpected place, causing it to fail.

Other tests take screenshots of 3d rendering windows for regression testing
against known "good" screenshots, and those are also susceptible to the
same problem, where a window poping up in front of another before a
screenshot might change the image and fail the test.

In summary, some of our tests can't be run in parallel with any other tests.

The current workaround is that we use a special mark in those tests that
can't run in parallel and run pytest twice: a session in parallel excluding
marked tests, and another regular, non-parallel session with only the
marked tests. This works, but it is error prone when developers run the
tests from the command line and makes error report cumbersome to read (the
first problem is alleviated a bit with a custom script).

Others have discussed other use cases in this thread:
https://bitbucket.org/hpk42/pytest/issue/175.

My PR takes care to distribute marked tests to a single node after all
other tests have executed, but after some discussion it is clear that there
might be more use cases out there so we would like to hear them before
deciding what would be the best interface options.

Cheers,

On Thu, Oct 23, 2014 at 11:48 AM, holger krekel <holger at merlinux.eu> wrote:

> Hi all,
>
> Currently, pytest-xdist in load-balancing ("-n4") does not support
> grouping of tests or influencing how tests are distributed to nodes.
> Bruno Oliveira did a PR that satisfies his company's particular
> requirements,
> in that GUI tests must run after all others and pinned to a particular
> note.
> (https://bitbucket.org/hpk42/pytest-xdist/pull-request/11/)
>
> My question is what needs do you have wrt to test distribution with xdist?
> I'd like to collect and write down some user stories before we design
> options/mechanisms and then possibly get to implement them.  Your input
> is much appreciated.
>
> best and thanks,
> holger
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20141023/39052479/attachment.html>