[issue30263] regrtest: log the system load?

[Jeremy Kloth]

Jeremy Kloth added the comment:

On Mon, Jun 19, 2017 at 6:24 AM, STINNER Victor <report at bugs.python.org> wrote:
>
> STINNER Victor added the comment:
>
>> Pushed utility to github:  https://github.com/jkloth/loadavg
>
> Hum, I see two implementations:
>
> * LOADAVG_USE_REG: use the registry, RegQueryValueExW(HKEY_PERFORMANCE_DATA, L"2", ...)
> * LOADAVG_USE_PDH: PdhOpenQueryW()
>
> Are these API accessible by everyone? On Linux, the data is public to everybody:

Yes.

> Why do you prefer registry over PDH?

A PDH query adds an additional 4MB to the running process.

> Should CalculateLoad be called every 5 seconds in a thread? If yes, IMHO it's a blocker issue for regrtest since some tests fail if other threads are running.

Yes, as that is exactly what Linux does (not in a thread, but as part
of the scheduler).  This is the reason for using the Thread Pool.
They are kernel provided worker threads for the running process.  As a
matter of fact, Windows 10 now uses them to speed up DLL loading on
startup so every Python process is running with 2+ threads.

However, regtest when run with -jX already starts an add'l thread so
that can no longer be an issue.

> I'm not sure that such code belongs to Python stdlib. What do you think of putting it on PyPI?

Sure, but packaging it up as a proper extension is quite low on my
todo list.  I am keen, however, on getting the buildbot fleet (well,
mine in particular) in good shape.

> For Python 2, sometimes I would like to use faulthandler on our buildbots since it's really hard to guess what crashed Python when a random test crash. But faulthandler was only added to Python 3.3. So using it on Python 2 would require to install it using pip. Right now, Python only requires itself to run tests. I'm not sure that it's a good idea to have dependencies, even if there are externals.

Agreed, which is why I offered to implement as a patch to the test suite.

> Again, I'm not sure that this specific issue is the best place to discuss implementing "os.getloadavg()", since your implementation is more complex than a single function call, it seems like it requires to run a thread "in background", so maybe an object to start/stop the thread, etc.

Don't get too caught up with `os.getloadavg()`, it was just a
possibility.  I was leaning more towards a private utility function
just for regrtest, but realized it may be useful outside of just
testing.  However, as you note, it does require a thread, which is why
it is best implemented using the Thread Pool API to minimize any
impact on Python proper.

----------
nosy: +jeremy.kloth

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue30263>
_______________________________________