[Python-Dev] tracking memory consumption

[Vladimir Marangozov]

Peter Funk wrote:
> 
> Hi!
> 
> In PR#214 Martin v. Loewis suggests a sizeof function as result of
> request to python-help.  I've followed the thread silently until now.

For such basic computations of the native objects sizes, struct.calcsize
is your friend.

> 
> On platforms with virtual memory subsystem this is usually a not an
> issue.  On embedded systems and ancient OSes (like MS-Dos) it is often
> useful, if applications can estimate how much memory their data consumes.
> 

Yes, we're moving slowly in this direction. The problem is not easy though,
because it has many, many facets: legacy code, garbage collection,
storage optimization, performance, etc.  So we need to think big here.

The roadmap (short & long term plan) that I'm preconizing is:

1. Take control over Python's memory.
   Presently, this is not the case; to remedy the situation, I've
   already sent a huge patch suite to python-patches aiming at gaining
   control over "the Python heap", without disturbing legacy code.

2. Collect some stats, based on 1, and possibly expose some of them
   to the user. Provide some mem & object monitoring facilities.

3. Optimize Python's mallocs (for instance, make the object-specific
   allocators aware of each other), based on the stats from 2.
   This will result in better mem sharing and possibly new speed/space
   tradeoffs (in conjuction with 4).

4. Take preventive actions, like garbage collection, mem compaction
   or other appropriate procedures, which will be trigerred on
   pertinent thresholds issued from 2. These are aimed at ensuring
   safety and liveness of the Python process.

Furthermore, I expect that the performance gain from 3) would be lost
for 4), so we'll probably end up with an added value, without performance
hits. This implies, of course, a number of compromises that need to be
weighted carefully, but we're not at this stage yet.

Currently, we can't do anything cool because we need 1) above all.
Gaining control on the Python heap is a precondition for any future work
on dynamic storage allocation/management.

-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252