Garbage collection

[Steve Holden]

Tom Wright wrote:
> Steven D'Aprano wrote:
>> You've described an extremely artificial set of circumstances: you create
>> 40,000,000 distinct integers, then immediately destroy them. The obvious
>> solution to that "problem" of Python caching millions of integers you
>> don't need is not to create them in the first place.
> 
> I know it's a very artificial setup - I was trying to make the situation
> simple to demonstrate in a few lines.  The point was that it's not caching
> the values of those integers, as they can never be read again through the
> Python interface.  It's just holding onto the space they occupy in case
> it's needed again.
> 
>> So what's your actual problem that you are trying to solve?
> 
> I have a program which reads a few thousand text files, converts each to a
> list (with readlines()), creates a short summary of the contents of each (a
> few floating point numbers) and stores this summary in a master list.  From
> the amount of memory it's using, I think that the lists containing the
> contents of each file are kept in memory, even after there are no
> references to them.  Also, if I tell it to discard the master list and
> re-read all the files, the memory use nearly doubles so I presume it's
> keeping the lot in memory.
> 
I'd like to bet you are keeping references to them without realizing it. 
The interpreter won't generally allocate memory that it can get by 
garbage collection, and reference counting pretty much eliminates the 
need for garbage collection anyway except when you create cyclic data 
structures.

> The program may run through several collections of files, but it only keeps
> a reference to the master list of the most recent collection it's looked
> at.  Obviously, it's not ideal if all the old collections hang around too,
> taking up space and causing the machine to swap.
> 
We may need to see code here for you to convince us of the correctness 
of your hypothesis. It sounds pretty screwy to me.

>>> but is there anything I can do to get that memory back without closing
>>> Python?
>> Why do you want to manage memory yourself anyway? It seems like a
>> horrible, horrible waste to use a language designed to manage memory for
>> you, then insist on over-riding it's memory management.
> 
> I agree.  I don't want to manage it myself.  I just want it to re-use memory
> or hand it back to the OS if it's got an awful lot that it's not using. 
> Wouldn't you say it was wasteful if (say) an image editor kept an
> uncompressed copy of an image around in memory after the image had been
> closed?
> 
Yes, but I'd say it was the programmer's fault if it turned out that the 
interpreter wasn't doing anything wrong ;-) It could be something inside 
an exception handler that is keeping a reference to a stack frame or 
something silly like that.

regards
  Steve
-- 
Steve Holden       +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd          http://www.holdenweb.com
Skype: holdenweb     http://del.icio.us/steve.holden
Recent Ramblings       http://holdenweb.blogspot.com