[Python-ideas] Add os.usable_cpu_count()

[Giampaolo Rodola']

On Tue, Sep 5, 2017 at 12:59 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Mon, Sep 4, 2017 at 8:59 PM, Giampaolo Rodola' <g.rodola at gmail.com>
> wrote:
> > Recently os.cpu_count() on Windows has been fixed in order to take
> process
> > groups into account and return the number of all available CPUs:
> > http://bugs.python.org/issue30581
> > This made me realize that os.cpu_count() does not return the number of
> > *usable* CPUs, which could possibly represent a better default value for
> > things like multiprocessing and process pools.
> > It is currently possible to retrieve this info on UNIX with
> > len(os.sched_getaffinity(0)) which takes CPU affinity and (I think) Linux
> > cgroups into account, but it's not possible to do the same on Windows
> which
> > provides this value via  GetActiveProcessorCount() API.
> > As such I am planning to implement this in psutil but would like to know
> how
> > python-ideas feels about adding a new os.usable_cpu_count() function (or
> > having os.cpu_count(usable=True)).
>
> This was discussed in https://bugs.python.org/issue23530
>
> It looks like the resolution at that time was:
>
> - os.cpu_count() should *not* report the number of CPUs accessible to
> the current process, but rather continue to report the number of CPUs
> that exist in the system (whatever that means in these days of
> virtualization... e.g. if you use KVM to set up a virtual machine with
> limited CPUs, then that changes os.cpu_count, but if you do it with
> docker then that doesn't).
>
> - multiprocessing and similar should continue to treat os.cpu_count()
> as if it returned the number of CPUs accessible to the current
> process.
>
> Possibly some lines got crossed there...
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>

I agree current os.cpu_count() behavior should remain unchanged. Point is
multiprocessing & similar are currently not taking CPU affinity and Linux
cgroups into account, spawning more processes than necessary.

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170905/38afd167/attachment-0001.html>