[issue43911] Queue.get() memory leak

[Raymond Hettinger]

Raymond Hettinger <raymond.hettinger at gmail.com> added the comment:

FWIW, when running your code on my Intel Mac with 16 Gb of RAM, the "after return" result is 0.1%.  So getting up to 2.2% seems to be specific to your build.

Also, I just ran your code with a deque instrumented to count the number of mallocs and frees.  At the end of the run, there were 312506 blocks allocated and 312490 freed.  The difference of 16 is expected.  That is the number of blocks kept in the freelist to reduce allocator calls for common use cases.

So we're not seeing a "leak" in the traditional sense of more calls to malloc() than to free().  Instead, there is just an indirect measurement of total process memory which is something we can't directly control.  

That you observed similar a result with a list of lists corroborates that there isn't an issue specific to Queue or deques.  Instead, it is just an artifact of this particular pattern of exercising the memory allocators.  

I conclude that what you're seeing isn't a bug.  The effect is due to a mix of:
1) the interaction between the C library and the O/S memory management routines
2) normally occurring fragmentation
3) freelists which are intentionally reserved.

Thank you for the report, but we've spent enough time on it.  Trying to account for total process memory is like chasing rainbows, mostly elusive and there is not pot of gold at the end :-)

----------
keywords: +patch
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed
Added file: https://bugs.python.org/file49974/instrument_deque.diff

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43911>
_______________________________________