[Patches] [ python-Patches-477059 ] __del__ on new classes vs. GC

[noreply@sourceforge.net]

Patches item #477059, was opened at 2001-10-31 20:13
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=477059&group_id=5470

Category: Core (C code)
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Guido van Rossum (gvanrossum)
Assigned to: Neil Schemenauer (nascheme)
Summary: __del__ on new classes vs. GC

Initial Comment:
Neil, excuse me for the assignment. You can reassign
this to Tim if you don't have time; I know *I* won't
get to it in time for 2.2b2.

I recently added __del__ for new-style classes, but I
believe this would require the same kind of exception
from the GC code as classic instances with a __del__
handler. Of course, it's a bit more difficult since
*any* heap-based type can have a __del__ handler...

----------------------------------------------------------------------

>Comment By: Guido van Rossum (gvanrossum)
Date: 2001-11-01 11:49

Message:
Logged In: YES 
user_id=6380

Neil, thanks for fixing my broken checkin. I think it still
needs a documentation change: the meaning of the variable
gc.garbage has changed.

In a discussion before lunch, Tim suggested that when Python
exits (or better, in Py_Finalize()) a check could be made if
there is any garbage in gc.garbage, and if so, a warning
about this should be printed to stderr. That seems a nice
feature.

In that same discussion, I realized that the module cleanup
behavior (where all globals in a module are set to None when
the module object is deallocated) is probably no longer
necessary now that we have GC.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-11-01 09:03

Message:
Logged In: YES 
user_id=31435

In Guido's example, if we assume the cycle will never ever 
get broken by the user, then reclaiming the X instances 
would be cool:  we know they're trash, and they're not 
themselves in a cycle.

The rub is that it's a bad assumption -- despite that the 
cycle is unreachable, it can still be gotten at, by the 
user, via gc's list of unreclaimable trash.  My model for 
what people should do is:

1. Don't create cycles with __del__ methods.

2. If you do, cleam 'em up yourself (and gc exposes
   the unreclaimable cycles, so that's quite doable even
   *after* the cycles become "unreachable").

So long as we expose unreclaimable cycles, nothing in a 
cycle is *truly* unreachable, and that makes deleting the X 
instances by magic dangerous (the user may break the cycle 
themself, and a destructor in one of the cycle members may 
need to access an intact X).

----------------------------------------------------------------------

Comment By: Neil Schemenauer (nascheme)
Date: 2001-11-01 08:45

Message:
Logged In: YES 
user_id=35752

We seem to have our messages passing each other in flight. :-)

In your example you don't see X.__del__ because the C
instances are never freed.  We can't call tp_clear on
either of the C instances because we can't choose an ordering
that ensures their __del__ methods will work.  For example, if
we clear "a" first but b.__del__ uses some attributes of "a"
then we will get an exception.  I suppose we could have an
option that says "call tp_clear on unreachable objects even
if they have finalizers".

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-11-01 08:39

Message:
Logged In: YES 
user_id=6380

I guess I misunderstood what you said before. Never mind.

----------------------------------------------------------------------

Comment By: Neil Schemenauer (nascheme)
Date: 2001-11-01 08:30

Message:
Logged In: YES 
user_id=35752

Anything with a finalizer (i.e. __del__ method) or reachable
from a finalizer does not get tp_clear called on it.  We
discussed this at length when the GC was being implemented.
I don't think anything has changed since then but I'm open
to more discussion.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-11-01 08:16

Message:
Logged In: YES 
user_id=6380

I'll leave this in your & Tim's capable hands.

I don't see any proof that tp_clear is being called; or
maybe there's no tp_clear for new-style instances. I ran
this script:

import gc

class C(object):
    def __del__(self): print "C.__del__"

class X(object):
    def __del__(self): print "X.__del__"

a = C()
a.x = X()
b = C()
b.x = X()
a.b = b
b.a = a
del a, b
gc.collect()

Before the change I checked in this morning (but after
adding __del__, obviously), this printed

X.__del__
C.__del__
C.__del__
X.__del__

Now it prints nothing.  I somehow had expected this

X.__del__
X.__del__

since the X instanes are not contributing to the cycle --
they are merely hanging off its sides. I probably
misunderstand something.

----------------------------------------------------------------------

Comment By: Neil Schemenauer (nascheme)
Date: 2001-11-01 07:50

Message:
Logged In: YES 
user_id=35752

This is not too serious of a problem and shouldn't be hard to
fix.  If we don't do anything then the result will be that
unreachable new-style classes will get tp_clear called on
them even if they have __del__ methods.  As Tim suggests,
fixing this should be a matter of removing some
PyInstance_Check tests.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-11-01 06:26

Message:
Logged In: YES 
user_id=6380

Thanks -- I thought it would probably shallow but wasn't
sure. I've fixed this now.

Do I understand correctly that objects with a __del__
attribute are not GC'ed at all (if part of a cycle)? Isn't
that sad? Shouldn't we at least attempt to recover the
memory without calling the finalizers?

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2001-10-31 22:13

Message:
Logged In: YES 
user_id=31435

Guido, I expect this is shallow, confined to move_finalizers
().  It's looking for objects that both (a) have 
a "__del__" attr, and (b) pass PyInstance_Check().  An 
instance of a new-style class with __del__ passes the 
former test but not the latter, and I believe that's all 
the repair needed (but am blanking out on exactly what 
should be checked instead -- it will occur to me overnight 
<wink> -- I assume the existing PyInstance_Check is really 
just trying to avoid the expense of general 
PyObject_HasAttr when the latter has no chance of 
succeeding).

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=477059&group_id=5470