[issue46205] Race condition in runtest_mp leads to hangs (never exits)

[Sam Gross]

New submission from Sam Gross <colesbury at gmail.com>:

The runtest_mp.py has a race condition between checking for worker.is_alive() and processing the queue that can lead to indefinite hangs.

The hang happens when the all the results from the self.output queue are processed but at least one of the workers hasn't finished exiting.

https://github.com/python/cpython/blob/8d7644fa64213207b8dc6f555cb8a02bfabeced2/Lib/test/libregrtest/runtest_mp.py#L394-L418

The main thread tries to get a result from the output queue, but the queue is empty and remains empty. Although the queue.get() operation eventually times out (after 30 seconds), the main thread does not re-check if all the workers have exited (!), but instead retries the queue.get() in the "while True" loop.

https://github.com/python/cpython/blob/8d7644fa64213207b8dc6f555cb8a02bfabeced2/Lib/test/libregrtest/runtest_mp.py#L415-L418

To reproduce, apply the below patch which introduces a small delay to more reliably trigger the hang.

curl "https://gist.githubusercontent.com/colesbury/fe3769f43dfb724c86ecbb182b1f6749/raw/e29a4eaeebb8d5252cdd66f3f8a70f7bc5fa14e7/runtest_mp.diff" | patch -p1
./python -m test test_regrtest -m test_module_from_test_autotest -v

----------
messages: 409374
nosy: colesbury
priority: normal
severity: normal
status: open
title: Race condition in runtest_mp leads to hangs (never exits)
versions: Python 3.10, Python 3.11, Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue46205>
_______________________________________