[New-bugs-announce] [issue26903] ProcessPoolExecutor(max_workers=64) crashes on Windows

[Diogo Pereira]

New submission from Diogo Pereira:

I'm using Python 3.5.1 x86-64 on Windows Server 2008 R2. Trying to run the ProcessPoolExecutor example [1] generates this exception:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Program Files\Python35\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python35\lib\concurrent\futures\process.py", line 270, in _queue_management_worker
    ready = wait([reader] + sentinels)
  File "C:\Program Files\Python35\lib\multiprocessing\connection.py", line 859, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Program Files\Python35\lib\multiprocessing\connection.py", line 791, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 64


The problem seems to be related to the value of the Windows constant MAXIMUM_WAIT_OBJECTS (see [2]), which is 64. This machine has 64 logical cores, so ProcessPoolExecutor defaults to 64 workers.

Lowering max_workers to 63 or 62 still results in the same exception, but max_workers=61 works fine.


[1] https://docs.python.org/3.5/library/concurrent.futures.html#processpoolexecutor-example
[2] https://hg.python.org/cpython/file/80d1faa9735d/Modules/_winapi.c#l1339

----------
components: Windows
messages: 264608
nosy: diogocp, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: ProcessPoolExecutor(max_workers=64) crashes on Windows
versions: Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26903>
_______________________________________