Reduce memory fragmentation with MALLOC_MMAP_THRESHOLD_ and MALLOC_MMAP_MAX_

[Lorenzo Bolla]

Hi all,

I've been experimenting with MALLOC_MMAP_THRESHOLD_ and MALLOC_MMAP_MAX_ env variables to affect memory management in a long-running Python 2 process.
See http://man7.org/linux/man-pages/man3/mallopt.3.html

I got the idea from this bug report: http://bugs.python.org/issue11849

The results I have are encouraging: memory fragmentation is reduced and the typical high-water mark visible in memory used by long-running processes is lower.

My only concern is if there are other side effects that may bite back, when using such low level tweaks. Does anyone have any experience in using them?

Here is an example script that shows how those variables affect RSS memory in a script that generate a large dictionary:
https://gist.github.com/lbolla/8e2640133032b0a6bb9c
Just run "alloc.sh" and compare the output.

Thanks,
L.