Limiting Python's Memory Use

[Johannes Stezenbach]

Vladimir Marangozov <Vladimir.Marangozov at inrialpes.fr> wrote:
>wpostma at my-deja.com wrote:
>> 
>> 1. Suppose I were to, one at a time over 24 hours, store rows into a
>> disk database, using an auto-incrementing integer key to the database.
>> Over 24 hours, the key value goes from 1 to over 200,000.  Would all
>> 200,000 "integer objects" be created as needed, and then freed, or are
>> the integer objects (200,000 of them) around for the duration of the
>> python interpreter's lifetime?
>> 
>
>The memory for holding 200,000 integers will be reserved for the duration
>of the interpreter's lifetime.

I think this needs clarification:

When you count a variable from 1 up to 200,000 each integer
object gets freed (added to the freelist) before (well, shortly
after in realitiy) the next one is created (from the freelist),
so you need only one (or two or three for temporary storage)
integer objects, not 200,000.

That's much different from creating all 200,000 at once, like in
range(200000).

>> 2. Suppose over 200,000 unique string constants were handled, are any of
>> them free'd or are they kept around? I saw a cache for commonly used
>> strings was created in the string object module, but I am unsure how
>> this cache is limited, in size.
>
>The cache is not limited in size. Some of the 200,000 unique strings
>will be interned, the others won't. "abc" will be interned, "a space"
>will not because it contains a space. The test for interning a string
>constant is that of strspn() relative to "0..9A..Z_a..z".
>This is done by the compiler.

Clarification:

Only string *constants* are candidates for interning. Strings
your script reads from input sources or somehow creates at
runtime will never be interned automatically.
(That's somehow equivalent to "This is done by the compiler".)

>Right. Python uses simple free lists for some of its objects for
>performance reasons: ints, small tuples, floats, frames. Simple free
>lists is the worst scenario for people with memory limits, not interested
>in performance, and with long running processes, because the free lists
>reserve storage and never release that storage + they degenerate in time.
>This is a killer for embedded systems with small memory. We're aware of
>this and we're working on it.

Free lists mean that not all memory that your script has used
will be freed. It means *not* that memory consumtion will grow
limitless because all garbage is kept around, just in case
someone could use it.
Some wasted memory usually won't be a problem for most (short or
long running) scripts.


Johannes