magic memory leak, Python or Windows problem?

[Tim Peters]

[Marcus Stojek]
> Im using Python 2.2 with Windows NT. After hunting a 'memory leak' for
> a couple of days my code is down to the following lines:
> #------------------------
> cont =1
> while cont:
>     lile = int(raw_input('listsize: '))
>     li=[]
>     for i in range(lile):

Note that range(lile) creates a list with lile integers in it.  This is a
key point.  The program should act very differently if you use xrange()
instead of range() here.

>         li.append(5.0)
>     stop = raw_input('press ENTER do delete list')
>     del li
>     wait = raw_input('again ?   ')
>     if wait[0] not in ['y','Y']:
>         cont=0
> #------------------------
> I am checking the memory usage of my prog with the Task-Manager Tool
> of NT. Starting the prog shows a memory usage of app. 2,200 KB. When
> generating a list of 2,000,000 floats mem increases to 34,900 KB.

The floats in this specific example don't really have anything to do with
it.  There is only 1 float object here (the one constructed for the literal
"0.5"), and this exact same float object gets appended to a list 2M times.
About 16 bytes total are used by that float object.

> Now, when deleting the list, the memory usage comes down to 27,000 KB.

35MB - 27MB ~= 8MB == 2M list entries times 4 bytes per list entry.  IOW,
the list space got reclaimed by the OS.  In general you can't know whether
it will or won't; in this case it was; see Martin's fine post for more on
that.

> Where did the remaining 25,000 KB go?

Believe it or not, it's sitting in the 2 million integer objects created by
range(2000000).  Each int object consumes 12 bytes on your box.  Python
doesn't even try to return memory for int objects to the system until it
shuts down; instead memory for int objects is saved in a dedicated
int-object free list, for quick reuse later.  Float objects have another,
similar dedicated free list, but you have only one float object in this
program.

> Repeating this procedure a second or third time, again ends up with
> 27,000 KB of mem usage. Only if I increase the size of my list, the
> memory usage after deleting the list increases.

Right, and the second time you do range(20000000) it goes faster, because
Python gets to reuse the space for the 20000000 integer objects it had to
create the first time you did it.

> Can anybody explain this?

Of course <wink>.

> Is this a problem of Pyhon or of what the Task Manager shows (or
> maybe of my brain)?

You don't have a real leak here, so it's a problem of your brain only if you
believe you do.  For the rest, Python ultimately gets all memory from the
platform C malloc()/free(), and C doesn't promise anything about when or
whether free()'ed memory will "be returned to the system".  Investigating
that is usually an excerise in uncovering undocumented accidents of the
specific platform you're running on (and, e.g., Microsoft's C
malloc()/free() act *very* differently across different Windows flavors, and
sometimes even across service packs).