Garbage collection

[Tom Wright]

Steven D'Aprano wrote:
> You've described an extremely artificial set of circumstances: you create
> 40,000,000 distinct integers, then immediately destroy them. The obvious
> solution to that "problem" of Python caching millions of integers you
> don't need is not to create them in the first place.

I know it's a very artificial setup - I was trying to make the situation
simple to demonstrate in a few lines.  The point was that it's not caching
the values of those integers, as they can never be read again through the
Python interface.  It's just holding onto the space they occupy in case
it's needed again.

> So what's your actual problem that you are trying to solve?

I have a program which reads a few thousand text files, converts each to a
list (with readlines()), creates a short summary of the contents of each (a
few floating point numbers) and stores this summary in a master list.  From
the amount of memory it's using, I think that the lists containing the
contents of each file are kept in memory, even after there are no
references to them.  Also, if I tell it to discard the master list and
re-read all the files, the memory use nearly doubles so I presume it's
keeping the lot in memory.

The program may run through several collections of files, but it only keeps
a reference to the master list of the most recent collection it's looked
at.  Obviously, it's not ideal if all the old collections hang around too,
taking up space and causing the machine to swap.

>> but is there anything I can do to get that memory back without closing
>> Python?
> 
> Why do you want to manage memory yourself anyway? It seems like a
> horrible, horrible waste to use a language designed to manage memory for
> you, then insist on over-riding it's memory management.

I agree.  I don't want to manage it myself.  I just want it to re-use memory
or hand it back to the OS if it's got an awful lot that it's not using. 
Wouldn't you say it was wasteful if (say) an image editor kept an
uncompressed copy of an image around in memory after the image had been
closed?

-- 
I'm at CAMbridge, not SPAMbridge