[pytest-dev] pytest overall execution timeout option

[Obermann, Stephan]

Just a thought, but if you want to kill the whole test run after 2h no matter what the actual test results are - why not simply set a timeout in your test automation system (the one that calls py.test). Most systems (Jenkins, Electric Commander, etc.) provide this feature and it would spare you from implementing your custom workflow logic on the "test runner" level.

Cheers,

/stephan

On 20 Apr 2016, at 14:08, Shankar Hiremath <shankarhiremath2006 at gmail.com<mailto:shankarhiremath2006 at gmail.com>> wrote:

Hi Bruno,

I tried with the below approach provided by you, when i increase the sleep time inside the test_foo to 10 sec,  then the test execution took 20.12 sec and then after further test case execution stooped.

but what i required is to stop the test itself instead of waiting for the test to complete (in this case 10 sec sleep), i agree for this case pytest-timeout is the best option.

My use case is little bit different: whenever there is a new commit, i am going to run smoke tests which are annotated with “pytest.mark.smoke” annotation in pytest (around 400 test cases)
when the new commit code quality is of good then all of the smoke tests will get completed well within 2 hours duration, when there is a product issue due to new commit randomly few of the tests might get hanged for some time due to that, the overall test execution will take > 2hours.

1) if all tests passed well within 2 hour duration i will say “+1” to the commit,
2) if any test failed & all are well within 2 hour duration all tests completed i will say “-1” to the commit,
3) if tests are taking more time >2 hours or any test hanged, i want to stop the current running test & stop further test execution, i will say “-1” to the commit,

Inorder to achieve the case-3, i need to have some mechanism of global timeout of 2 hours (i am ok with +/- 5 minutes difference accuracy) from pytest execution.
Any suggestion or approach to achieve this will very helpful. Thanks in advance.


def pytest_sessionstart(session):
    session.start_time = time.time()

@pytest.fixture(autouse=True)
def check_session_time(request):
    elapsed = time.time() - request.session.start_time
    if elapsed > 1.0:
        request.session.shouldstop = 'time limit reached: %0.2f seconds' % elapsed

@pytest.mark.parametrize('i', range(10))
def test_foo(i):
    time.sleep(10)

The output of pytest execution is as below:

================================================= test session starts =================================================
platform linux2 -- Python 2.7.6 -- py-1.4.31 -- pytest-2.6.2
plugins: timeout
collected 10 items

a.py ..

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: time limit reached: 10.02 seconds !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================== 2 passed in 20.12 seconds =============================================


Thanks
-Shankar


On Apr 19, 2016, at 9:22 PM, Shankar Hiremath <shankarhiremath2006 at gmail.com<mailto:shankarhiremath2006 at gmail.com>> wrote:

Hi Bruno,

Thanks a lot, this meets my requirement. I will go ahead with the approach provided by you.

Regards
-Shankar

On Mon, Apr 18, 2016 at 6:56 PM, Bruno Oliveira <nicoddemus at gmail.com<mailto:nicoddemus at gmail.com>> wrote:

Hi Shankar,

I don’t know of any plugin that does this out of the box, but the simplest way I found is to implement this is using an auto-use fixture:

# contents of conftest.py
import pytest
import time

def pytest_sessionstart(session):
    session.start_time = time.time()

@pytest.fixture(autouse=True)
def check_session_time(request):
    elapsed = time.time() - request.session.start_time
    if elapsed > 1.0:
        request.session.shouldstop = 'time limit reached: %0.2f seconds' % elapsed

# contents of test_foo.py
import pytest
import time

@pytest.mark.parametrize('i', range(10))
def test_foo(i):
    time.sleep(0.5)


Running this produces:

collected 10 items

test_foo.py ...

!!!!!!!!!!!!!!!! Interrupted: time limit reached: 1.11 seconds !!!!!!!!!!!!!!!!
========================== 3 passed in 1.69 seconds ===========================


Hope that helps.

Cheers,
Bruno.

?

On Mon, Apr 18, 2016 at 8:56 AM Shankar Hiremath <shankarhiremath2006 at gmail.com<mailto:shankarhiremath2006 at gmail.com>> wrote:
Hi All,

Is there any any plugin similar to "pytest-timeout” but at higher level (ex: complete suite level execution timeout)

My requirement is to run pytest with test suites and if all tests finishes well within the 2 hours time duration then its good, else I want to stop the test execution.
( I mean stop the current running test & further tests if any,  but the pytest call backs should get executed, for example: pytest_sessionfinish need to be executed, before exiting pytest)

Please suggest me if any existing plugins provides this feature, or any easy way to achieve this feature in pytest.

Thanks
-Shankar
_______________________________________________
pytest-dev mailing list
pytest-dev at python.org<mailto:pytest-dev at python.org>
https://mail.python.org/mailman/listinfo/pytest-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_pytest-2Ddev&d=CwMFaQ&c=lI8Zb6TzM3d1tX4iEu7bpg&r=azwxUlxNI1gzZVVp4XwBCw&m=gy4gz6ESbwBtwOGBjaOSY4AgSI3oyjOKc_Ozv4MhkAE&s=WoamsEXyLjxWkrVF7YmIQtQuSsody56tSM0Z4JKbz7g&e=>


_______________________________________________
pytest-dev mailing list
pytest-dev at python.org<mailto:pytest-dev at python.org>
https://mail.python.org/mailman/listinfo/pytest-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20160420/2d416500/attachment-0001.html>