Memory not being released?

[Steve M S Tregidgo]

Thanks for your response.  Some more of my own comments in reply:

gmcm at hypernet.com wrote:
> Steve Tregidgo writes:
> 
> > The actual data that results from the process is relatively small
> > (the useful stuff is less than 1MB, and other data -- such as
> > objects that have been dealt with, to prevent following circular
> > references -- is no more than a few MBs), and certainly well
> > below the size of the actual footprint.
> >
> > On the other hand, commenting out some other expressions (such as
> > those that remember data -- in other words, the things that I
> > would expect to take up memory and not release it again) doesn't
> > make a blind bit of difference.
> 
> That's a pretty good sign that you have circular references that 
> are keeping objects alive. Check out Cyclops from 
> ftp://ftp.python.org/pub/python/contrib/System/.
> 

Thanks, I'm looking at those file right now.

> Also, be aware that even if your python objects are getting 
> collected, that doesn't mean your c runtime is giving the 
> memory back to the OS. Although 250M would indicate a 
> pretty horrid c runtime.
>  
> > So what's going on?  My guesses so far have been along the lines
> > of refcounts not going down -- how else do I explain why memory
> > that was allocated was seemingly not deallocated?  I've tried
> > del'ing everything in sight, storing ids instead of objects,
> > inserting the odd sys.exc_traceback=None to clear those pesky
> > traceback reference holders ... none of it has helped so far.
> > 
> > Are there other hidden things that might hold references to my
> > objects?  And if so, what can I do about them?  After using an
> > object once, I don't want (don't need) to keep it in memory any
> > more, so for those objects I'll be revisiting at a later date, I
> > just remember their id and delete them.
> 
> In your code, dict1 is holding a reference to every object 
> you've visited, and thus keeping them all alive.

I'm storing obj.__id__ in each case, which is an integer -- I
certainly hope that it doesn't count as a reference to the object. 
Does it?  In mock-ups of the routine, storing a big list or
dictionary or numbers, the few MBs taken up by those structures were
not on a par with the size of the process in question.  Finally,
removing the lines (in the actual code, not the mock-up) which store
the dictionary didn't lower the footprint.  Frustrating, to say the
least!

> OTOH, judging 
> by "for item in obj.subs:", it looks like the whole graph is alive 
> all the time anyway. Essentially, you've got a huge pointer-
> based data structure. You need to hoist most of the smarts up 
> to a Graph object, and use some kind of persistent key to 
> identify objects (which are really on disk, or recreated on the 
> fly...). So that snippet might be:
>   for itemname in obj.subs:
>     obj = graph.getobject(itemname)
>  

This is something I considered -- at one stage in my attempts at
fixing the problem, I created a list of id's (so item.__id__ for each
item in obj.subs), deleted the copy of obj.subs (I was actually using
a list created by a method of obj, not an actual attribute), and then
retrieved a persistent item at the start of the loop, deleting it at
the end.  We do have persistent keys for objects, so I'm going to
take a look at this possibility again...

> > If it's any help, the process does something like this:
> > 
> > def recurse(obj, list, dict1, dict2):
> >   # Prevent circular recursion...
> >   if dict1.has_key(obj.__id__):
> >     return
> >   else:
> >     dict1[obj.__id__] = None
> > 
> >   a = do_thing(obj, dict2)
> > 
> >   # Recurse into obj's children, or do something else
> >   for item in obj.subs:
> >     if item.spam:
> >       recurse(item, list, dict1, dict2)
> >     else:
> >       b = do_thing(item, dict2)
> > 
> >   c = do_thing(obj, dict2)
> > 
> >   # Remember some things -- commenting
> >   # out produces no memory saving.
> >   list.append(a,b,c)
> > 
> > The function do_thing implements a loop; in total (over the whole
> > process) the loop's body is executed tens of thousands of times
> > -- it causes the biggest memory saving when commented out (and of
> > course it's the most important bit), but does no obvious
> > "remembering" of things.
> > 
> > 
> > Cheers,
> > Steve
> > 
> > --
> >        -- Steve Tregidgo --
> > Developer for Business Collaborator
> >         www.enviros.com/bc
> > 
> > 
> > Sent via Deja.com http://www.deja.com/
> > Before you buy.
> > 
> > -- 
> > http://www.python.org/mailman/listinfo/python-list
> 
> 
> 
> - Gordon