[Python-Dev] Discussion related to memory leaks requested

[Nick Coghlan]

On 15 January 2016 at 05:25, Matthew Paulson <paulson at busiq.com> wrote:

> Hi All:
>
> I've created a simple program to make sure I wasn't lying to you all ;->
>
> Here it is:
>
> for (ii = 0; ii < 100; ii++)
>         {
>             Py_Initialize();
>
>             if ((code = Py_CompileString(p, "foo", Py_file_input)) == NULL)
>                 printf("PyRun_SimpleString() failed\n");
>             else
>             {
>                 if (PyRun_SimpleString(p) == -1)
>                     printf("PyRun_SimpleString() failed\n");
>
>                 Py_CLEAR(code);
>             }
>
>             Py_Finalize();
>         }
>
> This sequence causes about 10k growth per iteration and after many cycles,
> there's no indication that any pooling logic is helping.  Our "useful"
> example is slightly more complex, and therefore may explain why I was
> seeing about 16k per iteration.
>
> Unless I've done something obviously wrong, I tend to believe Benjamin's
> claim that this issue is well known.
>
> Suggestion: I have had great success with similar problems in the past by
> using a pools implementation sitting on top of what I call a "block memory
> allocator".   The bottom (block) allocator grabs large blocks from the heap
> and then doles them out to the pools layer, which in turn doles them out to
> the requester.  When client memory is freed -- it is NOT -- rather it's
> added to the pool which contains like-sized blocks -- call it an "organized
> free list". This is a very, very fast way to handle high allocation
> frequency patterns.  Finally, during shutdown, the pool simply vaporizes
> and the block allocator returns a the (fewer) large blocks back to the
> heap.  This avoids thrashing the heap, forcing it to coalesce inefficiently
> and also avoids heap fragmentation, which can cause unwanted growth as
> well...
>
> Note that this would be a "hard-reset" of all allocated memory, and any
> global data in the text segment would also have to be cleared, but it would
> provide a fast, clean way to ensure that each invocation was 100% clean.
>

CPython does use an arena based allocator, but PyFinalize doesn't purge it
(if it did, there'd be segfaults rather than memory growth when modules
keep pointers across Initialize/Finalize cycles).

Building with PYMALLOC_DEBUG and setting PYTHONMALLOCSTATS in the
environment will cause it to dump debugging info during Py_Finalize.

Building with Py_TRACE_REFS and setting PYTHONDUMPREFS also provides info
on live Python objects during shutdown.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/0933176a/attachment.html>