[Python-Dev] Re: RFC: malloc cleanup

[Vladimir Marangozov]

[Detailed RFC on a bunch of malloc interfaces]

OK, I'll comment first as I'm probably one of the few who understands
(what I'm talking about :-) what's this mess is all about in its deepest
details while having an overall vision on the issue. So I feel obliged to
expose a more friendly explanation.

(This turned out to be only 2 screens long, so you may leave it for
 tomorrow morning ;-)

I'll make it simple:

What really happens if the RFC gets implemented and how it affects you?
-----------------------------------------------------------------------

1) All C extension modules, without exceptions, would be silently redirected
   to use the Python malloc wrappers. Silently means that no one line of code
   would need to be modified and the modules will continue to compile and run
   as before.  (i.e. all macros and functions used in C code remain the same)

   Consequences:

	a) You *won't* be required to modify anything, to change your programming
           habits or to adopt new mandatory interfaces. You could continue the same
           way as before.

	b) All *user-defined* Python objects (not the "core" ones, like ints, dicts
           or strings) would start using the wrappers automatically, which may result
           in a tiny degradation in their performance.

           This is the price to pay if we want to make Python user-malloc friendly --
           that is, decouple the core from its strong dependency on libc malloc
           and make 3rd party extensions depend only on Python's memory interfaces
           when they manipulate Python objects. Currently, the extensions depend
           on libc malloc (they use "public" memory offered by the system) and there's
           no way to make them use "private Python memory" for Python objects.

           BTW, user-defined objects are used far less frequently than the "core" ones.

2) All "core" objects will run as before at the same speed, but they'll use
   "private Python memory".

3) There would be an opportunity to make the code fancier. This is the whole point
   of the discussion about the NEW/DEL pairs. Fancier == more logical from the
   programmer's point of view. This is not mandatory, but it would be desirable
   to adopt these pairs so that mallocs don't mess again.

   And I'm volunteering to rename the core and the modules in the distribution
   so that they look fancier and that they serve as examples for future Python
   development. But all this remains optional.

4) Last, but not least, this will open the way to change (optionally) Python's
   core allocator to specialized and more appropriate mallocs than libc malloc.
   There's only one I know of for the moment (mine <wink>) and it's not "so good".
   But it looks promising, because it already serves Python better than libc malloc.

   The current state, however, prevents people to work in this area and to experiment
   alternatives, because it's very hard to get things working without an expert level
   knowledge of the internals. I personally find this regrettable and am willing to
   "repair" it, because I think that Python can make a noticeable profit from a
   specialized malloc (which may well outperform the tiny degradation caused by the
   wrappers).

------------

Currently I have a patch which does this the "easy way" -- it does not truly decouple
the core from the extensions through the wrappers. However, one can already make
experiments by recompiling Python *and* all user extensions with a new malloc. With
the patch, Python doesn't dump core anymore, which may be a little gain in your eyes,
but a huge one in mine <wink>.

FYI, I've corresponded in private with Neil Shemenauer who's working on GC, and he
tested & validated the present "easy" version of the patch by saying:

> On Sun, Feb 13, 2000 at 04:06:05PM +0100, Vladimir Marangozov wrote:
> > I'm appending another patch suite, so that you could test it directly
> > on your setup with your malloc. Let me know how it goes. Hope we're
> > making progress on this.
> 
> It works for me.  It passes the regression test and also does not
> leak memory when cycles are created. :)
> 
> 
>     Neil
> 

------------

So what I'm looking for here is your approval or your objections to the principles
I've exposed in this msg and in the RFC.

I can presently send a whole bunch of patched files affected by the "easy" way,
but it does not solve the real problem. The "easy way" does not redirect the
extensions to the wrappers. i.e. it doesn't have the "private Python memory" concept.
It *is* a step in the right direction, though.

Final word: I'm okay for deferring this issue for future Python releases, if there's
no time, no resources, no understanding, whatever (lately, I see a big increase in
Python's contributions since the announce of the tight schedule for 1.6). That's why
I won't push too much and I don't want to burded the atmosphere with still undecided
interfaces -- better take some more time and get them right. OTOH, the sooner we get
them right, the better.

-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252