[issue32986] multiprocessing, default assumption of Pool size unhelpful

[Matt Harvey]

Matt Harvey <m.j.harvey at acellera.com> added the comment:

Hi,

No, using the affinity's not useful to us as, in the general case, the batch system (PBS Pro in our case) isn't using cgroups or cpusets (it does control ave cpu use by monitoring rusage of the process group). 

Several other batch system I've worked with either set NCPUS directly or have a method for site-specific customisation of the job's environment.

That doesn't preclude using the affinity as an alternative to os.cpu_count()

As @pitrou correctly observes, probably better to have a simple, well-sign-posted way for the sysadmins to influence the pool default than try to overload multiprocessing with complex heuristics.

----------
nosy: +Matt Harvey

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue32986>
_______________________________________