[py-dev] performance timing and timer pausing in pytest-timeout

[Floris Bruynooghe]

Hi all,

While pytest-timeout was targeting hanging/deadlocked code there have
been various requests which relate to failing tests based on
performance.  Another issue which came up is the control of timeouts
during fixtures (funcarg/resource) setup.  So instead of saying that
this is not in the scope of pytest-timeout as I have been doing so far
here is a proposal of how to address these requirements.  I'd be very
grateful on any feedback you could provide.


Firstly I'd like to introduce a timeout funcarg resource which allows
you do to things like this::

   @pytest.funcarg()
   def shared_resource(timeout):
       with timeout.pause(), timeout.skip(30):
           return acquire_shared_resource()


Some details:

timeout.pause: context manager to pause the normal timeout timer if
one is active.

timeout.skip and timeout.fail: These are context managers which will
try to "soft" timeout the code block within them.  This means if
reasonably possible it will try to interrupt the code block after the
timeout, but if not it will just run the code block until it finished.
Either way when exiting the context manager either Skip or Fail will
be raised.  Also no extra information like e.g. thread and stack
details will be provided.  It's full signature will probably something
like timeout.skip(timeout, timer=time.time, msg=None) where the idea
is to provide some timers in the pytest namespace like
pytest.timer_ms, pytest.timer_pystone.  This can be useful in both
fixtures as well as test functions.


Lastly I'm thinking of providing a way of getting a timer which uses
pytest-cache and to measure the duration of a test (in pystones) and
will fail the test if it takes longer then the previous pystones
value.  This would only be reset by clearing the cache.  I'm not sure
whether to do this as part of the new timeout funcarg resource or as a
new marker,so either::

   def test_foo(timeout):
       with timeout.noregress():
           performance_critical_func()

or::

   @pytest.mark.timeout_noregress
   def test_foo():
       performance_coritical_func()

Both version could probably take a timer argument too re-using the
pytest.timer_ms, pytest.timer_pystones etc timers.

I'm slightly in favour of the context manager since it makes the timed
section more explicit.  But it might possibly only be allowed once per
test function (or have a "key" argument).


I'm very keen to hear other people's input on these ideas before
implementing them.  Especially since they solve problems I don't
really have personally.


Regards,
Floris