Python embedded in C & memory releasing

[Alex Martelli]

John J. Lee wrote:

> leonardbocock at yahoo.com (lebo) writes:
> 
>> I'm trying to understand how Python handles memory usage and dynamic
>> object loading and unloading.  Problem to solve? Build a very low
>> memory footprint (non-GUI) Python application for Unix/Windows. Why
>> use Python? End user flexibility.
>> 
>> So far it seems that (on Windows):
>> (1) memory increases when you import <module>. Expected behaviour.
>> (2) memory does not release when you del <module>. Why not?  Is Python
>> not releasing the memory or is it the OS not releasing memory?
> [...]
> 
> On almost all OSes, processes hold on to all memory that has been
> allocated by the OS.  Memory released by a process can only be reused
> *by that same process*.  The OS only gets it back when the process
> exits.

This is not true any more these days!!!  Consider the following program:

#include <stdio.h>
#include <stdlib.h>

int main()
{
        char* blobbone;

        printf("No blob yet, measure me now, then press enter... ");
        while(getchar() != '\n') {}
        printf("\n");

        blobbone = malloc(1024*1024);

        printf("I have the blob, measure me now, then press enter... ");
        while(getchar() != '\n') {}
        printf("\n");

        free(blobbone);

        printf("No more blob, measure me now, then press enter... ");
        while(getchar() != '\n') {}
        printf("\n");

        printf("Bye bye.\n");
        return 0;
}

Using Ctrl-Z to suspend and Enter to continue, on Linux, we see:

[alex at lancelot sae]$ gcc mem.c
[alex at lancelot sae]$ ./a.out
No blob yet, measure me now, then press enter...
[1]+  Stopped                 ./a.out
[alex at lancelot sae]$ ps -C a.out v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
32606 pts/6    T      0:00     73     1  1330  276  0.0 ./a.out
[alex at lancelot sae]$ fg
./a.out


I have the blob, measure me now, then press enter...
[1]+  Stopped                 ./a.out
[alex at lancelot sae]$ ps -C a.out v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
32606 pts/6    T      0:00     82     1  2362  320  0.0 ./a.out
[alex at lancelot sae]$ fg
./a.out


No more blob, measure me now, then press enter...
[1]+  Stopped                 ./a.out
[alex at lancelot sae]$ ps -C a.out v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
32606 pts/6    T      0:00     82     1  1334  316  0.0 ./a.out
[alex at lancelot sae]$ fg
./a.out


Bye bye.
[alex at lancelot sae]$

See?  the DRS grows from 1330 to 2362 K when the blob is allocated,
then drops back to 1334 when the blob is freed again.  I'm pretty
sure on Windows, with a good modern VC++ version, you'll see the
same, apart from the slightly higher difficulty of the "measure
me now" task;-).

And in Python...?  *SAME THING*!!!  Watch:

print "No blob yet, measure me now, then press enter...",
x = raw_input()
print

blobbone = 1024*256*[None]

print "I have the blob, measure me now, then press enter...",
x = raw_input()
print

del blobbone

print "No more blob, measure me now, then press enter...",
x = raw_input()
print

print "Bye bye."


[alex at lancelot sae]$ python mem.py
No blob yet, measure me now, then press enter...
[1]+  Stopped                 python mem.py
[alex at lancelot sae]$ ps -C python v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
32648 pts/6    T      0:00    436   779  3864 2504  0.4 python mem.py
[alex at lancelot sae]$ fg
python mem.py


I have the blob, measure me now, then press enter...
[1]+  Stopped                 python mem.py
[alex at lancelot sae]$ ps -C python v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
32648 pts/6    T      0:00    436   779  4892 3536  0.6 python mem.py
[alex at lancelot sae]$ fg
python mem.py


No more blob, measure me now, then press enter...
[1]+  Stopped                 python mem.py
[alex at lancelot sae]$ ps -C python v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
32648 pts/6    T      0:00    436   779  3864 2508  0.4 python mem.py
[alex at lancelot sae]$ fg
python mem.py


Bye bye.
[alex at lancelot sae]$

See?  Just the same behavior -- DRS grows 3864->4892, then shrinks
back to 3864 when the blob is dropped.


The explanation of the behavior observed by the original poster is
quite different from the one you posit: rather, it is that
"del module" does NOT drop the module -- the module remains in
memory, fully available and loaded, and it is pointed to by
the appropriate entry sys.modules['module'] in case you want it
back.  That's all there is to it -- modules, as opposed to, say,
lists, are NOT so easily disposed of:-).


Alex