[issue3526] Customized malloc implementation on SunOS and AIX

[Sébastien Sablé]

Sébastien Sablé <sable at users.sourceforge.net> added the comment:

Sorry for the very late reply; I have been quite busy recently with the 
birth of my second daughter, a new job, a new home town and soon a new home.

...
> But with your patch, such code wouldn't be thread-safe anymore. This
> patch implies that a thread can't call malloc directly or indirectly
> (printf, opendir, and many others) while it doesn't hold the GIL. This
> is going to break a lot of existing code.

I didn't have this problem since the threads in my application are 
handled by Python and so hold the GIL. But you are right it is a concern.

Fortunately, it is easy to solve by defining the following in dlmalloc:
#define HAVE_MORECORE 0

That way, all the memory allocations handled by Python will go in a 
dedicated mmaped memory segment controlled by dlmalloc, while all the 
calls to the system malloc will work as before (probably going into a 
segment handled by sbrk).

> It will also be slower, and consume more memory.

It should be noted that sbrk is deprecated on some platforms where mmap 
is suggested as a better replacement (Mac OS X, FreeBSD...).
sbrk is generally considered quite archaic.

I attach a new patch that can be applied to Python 2.7.1. It includes 
the dlmalloc modification and uses only mmap in this case (no sbrk).

We have delivered it in production with the new version of our software 
that works on AIX 6.1 and it works fine.

I also did some benchmarks and did not notice any slow down compared to 
a pristine Python 2.7.1 (actually it was slightly faster YMMV).
It also consumes a lot less memory, but that is the reason for this 
patch in the first place.

Since I am changing of job, I won't be working on AIX anymore (yeah!); I 
also don't expect this patch to be integrated spontaneously without 
someone interested in AIX pushing for it. So I leave this patch more as 
a reference for someone who would be impacted by this problem and would 
like to integrate it in his own Python. I hope it helps.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3526>
_______________________________________