[pytest-dev] xdist grouping: request for user stories

[Dj Gilcrease]

It isnt based on fixtures, though it could be. Right now it is just a hard
coded list of feature to node type mapping. We have ~15 different node
types and ~100 features. Right now we have a fixture that is a requirements
checker, and each test is tagged something like @pytest.marker.requires_abc
(it may have 1 to 20 of these).

The requirements checker grabs each test before it runs, checks all the
requires_X marks and does a check for each requirement by reading a file we
store on each node that has a list of all the features it supports if if
the requirement isnt in that list we skip the test. This has added a
significant amount of run time to our tests as we are running ~60k each run
on all 15 node types and about 10% of the tests will not run on one node
type or another. It would be nice if we could do the requirements
verification before we send the tests out to the node and only send tests
that the node can handle.


On Thu, Oct 23, 2014 at 1:32 PM, Bruno Oliveira <nicoddemus at gmail.com>
wrote:

> Hi Dj,
>
> How do you know beforehand if a slave satisfies a requirement or not? Is
> that based on fixtures? Can you provide more details, perhaps with some
> sample code to illustrate it better?
>
> One of the ideas mentioned by Holger would be the ability of controlling
> test distribution based on fixtures as well, so tests that use a certain
> fixture would automatically inherit those same properties. From what I
> could gather, perhaps this would be a good fit for your case.
>
> On Thu, Oct 23, 2014 at 4:30 PM, Dj Gilcrease <digitalxero at gmail.com>
> wrote:
>
>>
>> On Thu, Oct 23, 2014 at 7:46 AM, Bruno Oliveira <nicoddemus at gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> On Thu, Oct 23, 2014 at 11:48 AM, holger krekel <holger at merlinux.eu>
>>> wrote:
>>> > Bruno Oliveira did a PR that satisfies his company's particular
>>> requirements, in that GUI tests must run after all others and pinned to a
>>> particular note.
>>>
>>> To describe my requirements in a little more detail:
>>>
>>> We have a lot of GUI tests and most (say, 95%) work in parallel without
>>> any problems.
>>>
>>> A few of them though deal with window input focus, and those tests
>>> specifically can't run in parallel with other tests. For example, a test
>>> might:
>>>
>>> 1. create a widget and display it
>>> 2. edit a gui control, obtaining input focus
>>> 3. lose edit focus on purpose during the test, asserting that some other
>>> behavior occurs in response to that
>>>
>>> When this type of test runs in parallel with others, sometimes the other
>>> tests will pop up their own widgets and the first test will lose input
>>> focus at an unexpected place, causing it to fail.
>>>
>>> Other tests take screenshots of 3d rendering windows for regression
>>> testing against known "good" screenshots, and those are also susceptible to
>>> the same problem, where a window poping up in front of another before a
>>> screenshot might change the image and fail the test.
>>>
>>> In summary, some of our tests can't be run in parallel with any other
>>> tests.
>>>
>>> The current workaround is that we use a special mark in those tests that
>>> can't run in parallel and run pytest twice: a session in parallel excluding
>>> marked tests, and another regular, non-parallel session with only the
>>> marked tests. This works, but it is error prone when developers run the
>>> tests from the command line and makes error report cumbersome to read (the
>>> first problem is alleviated a bit with a custom script).
>>>
>>> Others have discussed other use cases in this thread:
>>> https://bitbucket.org/hpk42/pytest/issue/175.
>>>
>>> My PR takes care to distribute marked tests to a single node after all
>>> other tests have executed, but after some discussion it is clear that there
>>> might be more use cases out there so we would like to hear them before
>>> deciding what would be the best interface options.
>>>
>>> Cheers,
>>>
>>> On Thu, Oct 23, 2014 at 11:48 AM, holger krekel <holger at merlinux.eu>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Currently, pytest-xdist in load-balancing ("-n4") does not support
>>>> grouping of tests or influencing how tests are distributed to nodes.
>>>> Bruno Oliveira did a PR that satisfies his company's particular
>>>> requirements,
>>>> in that GUI tests must run after all others and pinned to a particular
>>>> note.
>>>> (https://bitbucket.org/hpk42/pytest-xdist/pull-request/11/)
>>>>
>>>> My question is what needs do you have wrt to test distribution with
>>>> xdist?
>>>> I'd like to collect and write down some user stories before we design
>>>> options/mechanisms and then possibly get to implement them.  Your input
>>>> is much appreciated.
>>>>
>>>> best and thanks,
>>>> holger
>>>> _______________________________________________
>>>> pytest-dev mailing list
>>>> pytest-dev at python.org
>>>> https://mail.python.org/mailman/listinfo/pytest-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> pytest-dev mailing list
>>> pytest-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pytest-dev
>>>
>>>
>>
>> I think it would be useful to pin or exclude marks from individual nodes.
>> We have marks like requires_abc, and right now we have logic that tests if
>> the requirement exists before running tests and marks them as skip if it is
>> not met, but I know beforehand which requirement can run on which nodes and
>> would like to not even send the tests that cannot run to a node.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20141023/22789519/attachment-0001.html>