[New-bugs-announce] [issue3526] Customized malloc implementation on SunOS and AIX

[Sébastien Sablé]

New submission from Sébastien Sablé <sable at users.sourceforge.net>:

Hi,

We run a big application mostly written in Python (with Pyrex/C
extensions) on different systems including Linux, SunOS and AIX.

The memory footprint of our application on Linux is fine; however we
found that on AIX and SunOS, any memory that has been allocated by our
application at some stage will never be freed at the system level.

After doing some analysis (see the 2 attached pdf documents), we found
that this is linked to the implementation of malloc on those various
systems:

The malloc used on Linux (glibc) is based on dlmalloc as described in
this document:
http://g.oswego.edu/dl/html/malloc.html

This implementation will use sbrk to allocate small chunks of memory,
but it will use mmap to allocate big chunks. This ensures that the
memory will actually get freed when free is called.

AIX and Sun have a more naive malloc implementation, so that the memory
allocated by an application through malloc is never actually freed until
the application leaves (this behavior has been confirmed by some experts
at IBM and Sun when we asked them for some feedback on this problem -
there is a 'memory disclaim' option on AIX but it is disabled by default
as it brings some major performance penalities).

For long running Python applications which may allocate a lot of memory
at some stage, this is a major drawback.

In order to bypass this limitation of the system on AIX and SunOS, we
have modified Python so that it will use the customized malloc
implementation dlmalloc like in glibc (see attached patch) - dlmalloc is
released in the public domain.

This patch adds a --enable-dlmalloc option to configure. When activated,
we observed a dramatic reduction of the memory used by our application.
I think many AIX and SunOS Python users could be interested by such an
improvement.

--
Sébastien Sablé
Sungard

----------
files: customized_malloc_SUN.pdf
messages: 70897
nosy: sable
severity: normal
status: open
title: Customized malloc implementation on SunOS and AIX
type: resource usage
Added file: http://bugs.python.org/file11082/customized_malloc_SUN.pdf

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3526>
_______________________________________