[pytest-dev] Fwd: Future of pytest-xdist/execnet?

Fri Oct 25 15:45:31 EDT 2019

Hi Isaul

Am 25.10.19 um 15:50 schrieb Isaul Vargas:
> This looks promising on the surface. I think we should outline what a 
> new concurrent test runner should look like.
> These are things in my wish list:
> * Tests are ran as a queue for each process. Currently x-dist batches 
> the tests to N workers, and maybe grabs a test or two in that batch. 
> But if any test takes a lot longer than expected, that batch is held up.
We intend to add richer sheduling patters (which in turn will need richr 
communication about collection (marks, fixtures, parameters)
> * The ability to pin tests to a worker. Maybe you have a class of 
> tests that need a single shared resource that would be difficult to 
> pickle, so using a marker allows you to do this. There should be 
> markers to allow grouping of tests to a single worker. Probably at the 
> class and module level.
dito
> * Add tooling to make it easy for fixtures to share data among 
> processes. Maybe a fixture runs a long process to generate data, but 
> the data is easily pickleable. There should be way to have a fixture 
> initialize once, and share the result with other processes.
we will introduce something to detail  shared resources, but we want 
make fixtures that are inter-process, its just not possible to get 
sensibly right
> * Allow some form of locking for function scope fixtures so that they 
> can block when running concurrently. This could be useful for things 
> that are complicated to share, but you want only one process to run at 
> at time during setup or teardown.

i believe a resource hand-off protocol can be integrated with scheduling,
this detail will need careful design work


-- Ronny


>
>
> I'll be happy to test this new project and provide feedback. I am 
> really excited that is happening.
>
> On Thu, Oct 24, 2019 at 8:46 AM Bruno Oliveira <nicoddemus at gmail.com 
> <mailto:nicoddemus at gmail.com>> wrote:
>
>     Hi everyone,
>
>     For some time now |execnet| has been in maintenance only mode, and
>     even so very few people are willing to maintain it; lately just
>     myself and I’m not a good choice given that I don’t know the
>     codebase at all, plus I have tons on my plate already. This poses
>     a problem because often we have bug reports or feature requests in
>     |pytest-xdist| which would require fixing or improving |execnet|,
>     and are currently left without solution.
>
>     Ronny and I have on occasion discussed the possibility of changing
>     |execnet|‘s backend , with the current contenders being
>     |multiprocessing| for local distribution and |mitogen| for remote
>     distribution.
>
>     I would like to write down some thoughts and gather opinions from
>     the mailing list members.
>
>
>         multiprocessing
>
>     Aside from not needing to maintain |execnet| anymore, being able
>     to use |multiprocessing| would bring several benefits:
>
>      *
>
>         |pytest -s| would work just fine, because |multiprocessing|
>         does not use |stdout/stderr| for communication. This is a
>         often requested feature but which we can’t (with current
>         expertise) implement on |execnet|.
>
>      *
>
>         It would automatically support anything that can be picked to
>         be transmitted across the wire. Currently execnet does not
>         support custom user types and some builtins (frozen sets for
>         example).
>
>     Without |execnet| we would lose the ability of running the tests
>     in different interpreter versions, but I believe this has been
>     supplanted by |tox| in the ecosystem.
>
>
>         mitogen
>
>     Using |mitogen| for remote execution would provide automatic
>     bootstrap of the Python environment, which is not currently
>     supported by |xdist|‘s ssh remote execution, greatly limiting its
>     usefulness in real-time scenarios.
>
>     This is Ronny’s idea and he can add more here if wanted.
>
>
>         pytest-xdist2?
>
>     All the changes mentioned above would break backward compatibility
>     *hard*, because details of the |execnet| implementation are
>     currently exposed to users in the command-line:
>
>     |pytest -d --tx popen//python |
>
>     And in several hooks:
>
>     |def pytest_xdist_newgateway(gateway): """ called on new raw
>     gateway creation. """ |
>
>     (|gateway| is an |execnet| object)
>
>     To incorporate the new backends, we would need to severely break
>     both command-line and existing hooks. Given the level of breakage
>     this could cause, just bumping the major version in this case
>     doesn’t seem enough, it would be a disservice to users given that
>     probably nobody pins pytest-xdist to |<=2|, and it would break the
>     world with hard to understand error messages once a first major
>     release was released.
>
>     Because of this, we have been discussing creating a new package,
>     |pytest-xdist2| (other suggestions are welcome), without any
>     backward compatibility guarantees with |pytest-xdist|.
>
>     |pytest-xdist2| would, at first, only support local test
>     distribution using |multiprocessing| (|-n| flag), no new hooks,
>     and no remote execution. I’ve made a proof of concept here:
>
>     https://github.com/pytest-dev/pytest-xdist/pull/479
>
>     So it definitely seems possible. One immediate benefit is that the
>     proof of concept above supports |pytest -s| already, and can
>     transfer anything that’s pickled over the wire.
>
>     So I would like to know what people think of the above ideas, if
>     they are good/bad, and/or suggest alternatives that we are not
>     seeing right now.
>
>     Ronny, please feel free to add to this email and correct anything
>     I got wrong.
>
>     Cheers,
>     Bruno
>
>     _______________________________________________
>     pytest-dev mailing list
>     pytest-dev at python.org <mailto:pytest-dev at python.org>
>     https://mail.python.org/mailman/listinfo/pytest-dev
>
>
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20191025/abdd3fa7/attachment-0001.html>