[pytest-dev] Future of pytest-xdist/execnet?

[Bruno Oliveira]

Hi everyone,

For some time now execnet has been in maintenance only mode, and even so
very few people are willing to maintain it; lately just myself and I’m not
a good choice given that I don’t know the codebase at all, plus I have tons
on my plate already. This poses a problem because often we have bug reports
or feature requests in pytest-xdist which would require fixing or improving
execnet, and are currently left without solution.

Ronny and I have on occasion discussed the possibility of changing execnet‘s
backend , with the current contenders being multiprocessing for local
distribution and mitogen for remote distribution.

I would like to write down some thoughts and gather opinions from the
mailing list members.
multiprocessing

Aside from not needing to maintain execnet anymore, being able to use
multiprocessing would bring several benefits:

   -

   pytest -s would work just fine, because multiprocessing does not use
   stdout/stderr for communication. This is a often requested feature but
   which we can’t (with current expertise) implement on execnet.
   -

   It would automatically support anything that can be picked to be
   transmitted across the wire. Currently execnet does not support custom user
   types and some builtins (frozen sets for example).

Without execnet we would lose the ability of running the tests in different
interpreter versions, but I believe this has been supplanted by tox in the
ecosystem.
mitogen

Using mitogen for remote execution would provide automatic bootstrap of the
Python environment, which is not currently supported by xdist‘s ssh remote
execution, greatly limiting its usefulness in real-time scenarios.

This is Ronny’s idea and he can add more here if wanted.
pytest-xdist2?

All the changes mentioned above would break backward compatibility *hard*,
because details of the execnet implementation are currently exposed to
users in the command-line:

pytest -d --tx popen//python

And in several hooks:

def pytest_xdist_newgateway(gateway):
    """ called on new raw gateway creation. """

(gateway is an execnet object)

To incorporate the new backends, we would need to severely break both
command-line and existing hooks. Given the level of breakage this could
cause, just bumping the major version in this case doesn’t seem enough, it
would be a disservice to users given that probably nobody pins pytest-xdist
to <=2, and it would break the world with hard to understand error messages
once a first major release was released.

Because of this, we have been discussing creating a new package,
pytest-xdist2 (other suggestions are welcome), without any backward
compatibility guarantees with pytest-xdist.

pytest-xdist2 would, at first, only support local test distribution using
multiprocessing (-n flag), no new hooks, and no remote execution. I’ve made
a proof of concept here:

https://github.com/pytest-dev/pytest-xdist/pull/479

So it definitely seems possible. One immediate benefit is that the proof of
concept above supports pytest -s already, and can transfer anything that’s
pickled over the wire.

So I would like to know what people think of the above ideas, if they are
good/bad, and/or suggest alternatives that we are not seeing right now.

Ronny, please feel free to add to this email and correct anything I got
wrong.

Cheers,
Bruno
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20191024/662093f6/attachment.html>