[New-bugs-announce] [issue45945] compileall.py throws a traceback when using -j0 and thus 'make install' locks up

[Alexander Kanavin]

New submission from Alexander Kanavin <alex.kanavin at gmail.com>:

Hello, Yocto project has had to disable -j0 for compileall, so that it runs serially. If it doesn't, then 'make install' locks up sporadically, with hanging processes:
```
3837320 ?        SN     0:00                  \_ make -j 16 -l 52 STAGING_LIBDIR=/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/recipe-sysroot/usr/lib STAGING_INCDIR=/
 157523 ?        SNl    0:02                      \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -d /u
 160673 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160677 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160682 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160697 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160759 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
 160816 ?        SN     0:00                          \_ python3.10 -Wi -OO /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280438/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py -j0 -
...
```

and installation log reveals:
```
poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/compileall.py \
        -j0 -d /usr/lib/python3.10 -f \
        -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
        /home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/asyncio'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/collections'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/config-3.10-x86_64-linux-gnu'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/ctypes'...
Listing '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/ctypes/macholib'...
...
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures/process.py", line 317, in run
    result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures/process.py", line 376, in wait_result_broken_or_wakeup
    worker_sentinels = [p.sentinel for p in self.processes.values()]
  File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1280388/tmp/work/core2-64-poky-linux/python3/3.10.0-r0/image/usr/lib/python3.10/concurrent/futures/process.py", line 376, in <listcomp>
    worker_sentinels = [p.sentinel for p in self.processes.values()]
RuntimeError: dictionary changed size during iteration
```

----------
components: Library (Lib)
messages: 407431
nosy: Alexander Kanavin
priority: normal
severity: normal
status: open
title: compileall.py throws a traceback when using -j0 and thus 'make install' locks up
versions: Python 3.10

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45945>
_______________________________________