[pytest-dev] Fwd: Future of pytest-xdist/execnet?

[Isaul Vargas]

This looks promising on the surface. I think we should outline what a new
concurrent test runner should look like.
These are things in my wish list:
* Tests are ran as a queue for each process. Currently x-dist batches the
tests to N workers, and maybe grabs a test or two in that batch. But if any
test takes a lot longer than expected, that batch is held up.
* The ability to pin tests to a worker. Maybe you have a class of tests
that need a single shared resource that would be difficult to pickle, so
using a marker allows you to do this. There should be markers to allow
grouping of tests to a single worker. Probably at the class and module
level.
* Add tooling to make it easy for fixtures to share data among processes.
Maybe a fixture runs a long process to generate data, but the data is
easily pickleable. There should be way to have a fixture initialize once,
and share the result with other processes.
* Allow some form of locking for function scope fixtures so that they can
block when running concurrently. This could be useful for things that are
complicated to share, but you want only one process to run at at time
during setup or teardown.


I'll be happy to test this new project and provide feedback. I am really
excited that is happening.

On Thu, Oct 24, 2019 at 8:46 AM Bruno Oliveira <nicoddemus at gmail.com> wrote:

> Hi everyone,
>
> For some time now execnet has been in maintenance only mode, and even so
> very few people are willing to maintain it; lately just myself and I’m not
> a good choice given that I don’t know the codebase at all, plus I have tons
> on my plate already. This poses a problem because often we have bug reports
> or feature requests in pytest-xdist which would require fixing or
> improving execnet, and are currently left without solution.
>
> Ronny and I have on occasion discussed the possibility of changing execnet‘s
> backend , with the current contenders being multiprocessing for local
> distribution and mitogen for remote distribution.
>
> I would like to write down some thoughts and gather opinions from the
> mailing list members.
> multiprocessing
>
> Aside from not needing to maintain execnet anymore, being able to use
> multiprocessing would bring several benefits:
>
>    -
>
>    pytest -s would work just fine, because multiprocessing does not use
>    stdout/stderr for communication. This is a often requested feature but
>    which we can’t (with current expertise) implement on execnet.
>    -
>
>    It would automatically support anything that can be picked to be
>    transmitted across the wire. Currently execnet does not support custom user
>    types and some builtins (frozen sets for example).
>
> Without execnet we would lose the ability of running the tests in
> different interpreter versions, but I believe this has been supplanted by
> tox in the ecosystem.
> mitogen
>
> Using mitogen for remote execution would provide automatic bootstrap of
> the Python environment, which is not currently supported by xdist‘s ssh
> remote execution, greatly limiting its usefulness in real-time scenarios.
>
> This is Ronny’s idea and he can add more here if wanted.
> pytest-xdist2?
>
> All the changes mentioned above would break backward compatibility *hard*,
> because details of the execnet implementation are currently exposed to
> users in the command-line:
>
> pytest -d --tx popen//python
>
> And in several hooks:
>
> def pytest_xdist_newgateway(gateway):
>     """ called on new raw gateway creation. """
>
> (gateway is an execnet object)
>
> To incorporate the new backends, we would need to severely break both
> command-line and existing hooks. Given the level of breakage this could
> cause, just bumping the major version in this case doesn’t seem enough, it
> would be a disservice to users given that probably nobody pins pytest-xdist
> to <=2, and it would break the world with hard to understand error
> messages once a first major release was released.
>
> Because of this, we have been discussing creating a new package,
> pytest-xdist2 (other suggestions are welcome), without any backward
> compatibility guarantees with pytest-xdist.
>
> pytest-xdist2 would, at first, only support local test distribution using
> multiprocessing (-n flag), no new hooks, and no remote execution. I’ve
> made a proof of concept here:
>
> https://github.com/pytest-dev/pytest-xdist/pull/479
>
> So it definitely seems possible. One immediate benefit is that the proof
> of concept above supports pytest -s already, and can transfer anything
> that’s pickled over the wire.
>
> So I would like to know what people think of the above ideas, if they are
> good/bad, and/or suggest alternatives that we are not seeing right now.
>
> Ronny, please feel free to add to this email and correct anything I got
> wrong.
>
> Cheers,
> Bruno
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20191025/b35d6f59/attachment-0001.html>