[issue45077] multiprocessing.Pool(64) crashes on Windows

[Eryk Sun]

Eryk Sun <eryksun at gmail.com> added the comment:

See bpo-26903 for a similar problem in concurrent.futures.ProcessPoolExecutor. It was resolved by adding a limit constant, _MAX_WINDOWS_WORKERS == 61. 

WaitForMultipleObjects() can wait on up to 64 object handles, but in this case 3 slots are already taken. The pool wait includes two events for its output and change-notifier queues (named pipes), plus the _winapi module always reserves a slot for the SIGINT event, even though this event is only used by waits on the main thread.

To avoid the need to limit the pool size, connection._exhaustive_wait() could be modified to combine simultaneous waits on up to 63 threads, for which each thread exhaustively populates a list of up to 64 signaled objects. I wouldn't want to modify _winapi.WaitForMultipleObjects, but the exhaustive wait should still be implemented in C, probably in the _multiprocessing extension module. A benefit of implementing _exhaustive_wait() in C is lightweight thread creation, directly with CreateThread() and a relatively small stack commit size.

----------
components: +Library (Lib)
nosy: +eryksun
versions: +Python 3.10, Python 3.11

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45077>
_______________________________________