memory allocation

[Tim Peters]

[Tim Peters]
>> You can build Python without pymalloc.  I've never done that myself,
>> so am not sure how to do it.  Could be that passing --with-pymalloc=no
>> would suffice.

[benevilent at optusnet.com.au]
> I have built with this option, but to no avail.

Then the Python you built was not using pymalloc.  I suppose you could
confirm that by setting breakpoints in pymalloc and noting that
they're never hit (or noting that the debugger won't let you set
breakpoints there anymore, because the pymalloc code no longer
exists!).

> I can provide a sample c program (which has it's virtual memory size expand
> and contract dynamically), and a similar python program for which this does
> not occur.

If you're not using pymalloc anymore, then you're trying to out-guess
the behavior of your specific platform C's malloc/free system, and how
it interacts with your specific platform's operating system.  There's
no guarantee that "returning" memory to the platform free() has any
effect on "the memory used by the program" as reported by the OS. 
Indeed, if you have turned pymalloc off, Python itself is just another
C program using malloc and free.  I'm not interested in staring at
your platform C+OS, but I'll note that it's common as mud, across many
platforms, to return memory to free() and see no effect on reported VM
size.  As I said before, the behavior here is typically very complex,
depending on exact traces of when malloc() and free() are called, on
the exact arguments passed to them, and on myriad details of how
malloc/free are implemented on your platform.  It's not actually
interesting to find a C program where "it works", the interesting bit
is finding C programs where "it doesn't work", because the latter show
the platform's weaknesses.

A common pattern, extended to exhaustion to make a point:

     malloc(big_number)
     malloc(little_number)
     malloc(big_number)
     malloc(little_number)
     malloc(big_number)
     malloc(little_number)
 ...
     repeat until you run out of VM

This can fragment address space so badly that returning *all* the
memory allocated by the "big_number" calls nevertheless leaves
allocated little pieces scattered across the entire virtual address
space.  So long as the little pieces remain allocated, there's nothing
that even a fiercely determined malloc/free could do to return any
part of the address space to the OS.  In real life, malloc/free
typically don't try hard at all to return space to the OS, burning
just a few cycles after a free() to see whether it's obviously
possible to do.  That's easily frustrated.

Depending on your platform C library, you may or may not be able to
ask its malloc to show you a map of allocated and available regions. 
If you can, you'll probably find a fragmented address space.  Figuring
out "why" it's fragmented is another big job, and then it may or may
not be possible to rearrange computations to avoid the cause(s)
discovered.

There's rarely an easy answer to one of these.