[issue39763] distutils.spawn should use subprocess (hang in parallel builds on QNX)
STINNER Victor
report at bugs.python.org
Mon Mar 2 07:22:43 EST 2020
STINNER Victor <vstinner at python.org> added the comment:
> And setup.py does use use multiple processes via a thread pool, where each thread calls fork(), if it detects that "make" was invoked with the parallel (-j) option.
Oh ok, now I get it :-) setup.py uses distutils to build extensions and distutils.command.build_ext uses concurrent.futures for parallelism:
if self.parallel:
self._build_extensions_parallel()
else:
self._build_extensions_serial()
with:
def _build_extensions_parallel(self):
workers = self.parallel
if self.parallel is True:
workers = os.cpu_count() # may return None
try:
from concurrent.futures import ThreadPoolExecutor
except ImportError:
workers = None
if workers is None:
self._build_extensions_serial()
return
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = [executor.submit(self.build_extension, ext)
for ext in self.extensions]
for ext, fut in zip(self.extensions, futures):
with self._filter_build_errors(ext):
fut.result()
The problem is not concurrent.futures but the job submitted to the executor: build_extension() uses distutils.spawn() which is unsafe.
distutils.spawn() is the root issue if I understood correctly. It must be rewritten with subprocess.
----------
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39763>
_______________________________________
More information about the Python-bugs-list
mailing list