[New-bugs-announce] [issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity

[Julian Taylor]

New submission from Julian Taylor:

multiprocessing.cpu_count and os.cpu_count which are often used to determine how many processes one can run in parallel do not respect the cpuset which may limit the process to only a subset of online cpus leading to heavy oversubscription in e.g. containerized environments:

$ taskset -c 0 python3.4 -c 'import multiprocessing; print(multiprocessing.cpu_count())'
32
$ taskset -c 0 python3.4 -c 'import os; print(os.cpu_count())'
32

While the correct result here should be 1.

This requires programs to have to use less portable methods like forking to gnu nproc or having to read the /proc filesystem.

Having a keyword argument to switch between online and available cpus would be fine too.

----------
components: Library (Lib)
messages: 236671
nosy: jtaylor
priority: normal
severity: normal
status: open
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23530>
_______________________________________