[Python-Dev] Have a big machine and spare time? Here's a possible Python bug.

[Thomas Wouters]

On Thu, May 23, 2019 at 5:15 PM Steve Dower <steve.dower at python.org> wrote:

> On 23May2019 0542, Inada Naoki wrote:
> > 1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel
> space.
> > 2. This loop is cleary hot:
> >
> https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819
> >
> > I can attach the process by gdb and I confirmed many arenas have
> > same nfreepools.
>
> It's relatively easy to test replacing our custom allocators with the
> system ones, yes? Can we try those to see whether they have the same
> characteristic?
>
> Given the relative amount of investment over the last 19 years [1], I
> wouldn't be surprised if most system ones are at least as good for our
> needs now. Certainly Windows HeapAlloc has had serious improvements in
> that time to help with fragmentation and small allocations.
>

FYI, and I've mentioned this at PyCon to a few people (might've been you,
Steve, I don't remember) -- but at Google we've experimented with disabling
obmalloc when using TCMalloc (a faster and thread-aware malloc, which makes
a huge difference within Google when dealing with multi-threaded C++
libraries), both using the usual Python benchmarks and real-world code with
real-world workloads (a core part of YouTube, for example), all on Linux.
There's still a significant benefit to using obmalloc when using glibc's
malloc, and also a noticeable benefit when using TCMalloc. There are
certainly cases where it doesn't matter much, and there may even be cases
where the overhead of obmalloc isn't worth it, but I do believe it's still
a firm net benefit.


>
> Cheers,
> Steve
>
> [1]:
>
> https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L769
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/thomas%40python.org
>


-- 
Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190524/f9ba4546/attachment.html>