Weakref.ref callbacks and eliminating __del__ methods

[Tim Peters]

[Mike C. Fletcher]
> I'm looking at rewriting parts of Twisted and TwistedSNMP to eliminate
> __del__ methods (and the memory leaks they create).

A worthy goal!

> Looking at the docs for 2.3's weakref.ref, there's no mention of whether the
> callbacks are held with a strong reference.

A callback is strongly referenced from the weakref(s) containing it.

>  My experiments suggest they are not... i.e. I'm trying to use this pattern:
>
> class Closer( object ):
>    """Close the OIDStore (without a __del__)"""
>    def __init__( self, btree ):
>        """Initialise the closer object"""
>        self.btree = btree
>    def __call__( self, oldObject=None ):
>        """Regular call via self.close or weakref deref"""
>        if self.btree:
>            self.btree.close()
>            self.btree = None
> class BSDOIDStore(oidstore.OIDStore):
>    def __init__( self, filename, OIDs = None ):
>        """Initialise the storage with appropriate OIDs"""
>        self.btree = self.open( filename )
>        self.update( OIDs )
>        self.close = Closer( self.btree )
>        weakref.ref( self, self.close )
> 
> but the self.close reference in the instance is going away *before* the
> object is called.

It would really help to give a minimal example of self-contained
executable code.  I'm not sure how you're using this code, and
guessing takes too long.

My best brief guess is that it's equivalent to this:

>>> class A: pass
>>> a = A()
>>> def f(*args):
...     print "callback called"
>>> import weakref
>>> weakref.ref(a, f)
<weakref at 00BB2570; to 'instance' at 00B8F878>
>>> 2
2
>>> del a

-- and the callback isn't called --

The "2" there is important, just because this is an interactive
session.  The point of it was to stop `_` from holding a reference to
the created weakref.  In your code

        weakref.ref( self, self.close )

the weakref itself becomes trash immediately after it's created, so
the weakref is thrown away entirely (including its strong reference to
self.close, and its weak reference to self) right after that line
ends.  Of course callbacks aren't *supposed* to be called when the
weakref itself goes away, they're supposed to be called when the thing
being weakly referenced goes away.  So this all looks vanilla and
inevitable to me -- the callback is never invoked because the weakref
itself goes away long before self goes away.

> So, this approach doesn't *seem* to work (the Closer doesn't get
> called), so I can gather that the callbacks don't get incref'd (or they
> get decref'd during object deletion).

I'm pretty sure it's just because there's nothing here to keep the
weakref itself alive after it's created.  You could try, e.g.,

    self.wr = weakref.ref( self, self.close )

to keep it alive, but things get fuzzy then if self becomes part of
cyclic trash.

> I can work around it in this particular case by defining a __del__ on
> the Closer, but that just fixes this particular instance (and leaves
> just as many __del__'s hanging around).  I'm wondering if there's a
> ready recipe that can *always* replace a __del__'s operation?

In the presence of cycles it gets tricky; there's a lot of
more-or-less recent words (less than a year old <wink>) about that on
python-dev.  It gets complicated quickly, and it seems nobody can make
more time to think about it.

> I know I heard a rumour somewhere about Uncle Timmy wanting to eliminate
> __del__ in 2.5 or thereabouts,

Python 3 at the earliest.  That's the earliest everything nobody can
make time for lands <0.5 wink>.

> so I gather there must be *some* way of handling the problem generally.  The
> thing is, weakref callbacks trigger *after* the object is deconstructed, while
> __del__ triggers before...
> must be something clever I'm missing.
>
> Throw an old doggie a bone?

There are unresolved issues about how to get all this stuff to work
sanely.  The driving principle behind cyclic-trash weakref endcases
right now is "do anything defensible that won't segfault".

I'll note that one fairly obvious pattern works very well for weakrefs
and __del__ methods (mutatis mutandis):  don't put the __del__ method
in self, put it in a dead-simple object hanging *off* of self.  Like
the simple:

class BTreeCloser:
    def __init__(self, btree):
        self.btree = btree

    def __del__(self):
        if self.btree:
            self.btree.close()
            self.btree = None

Then give self an attribute refererring to a BTreeCloser instance, and
keep self's class free of a __del__ method.  The operational
definition of "dead simple" is "may or may not be reachable only from
cycles, but is never itself part of a cycle".  Cyclic gc won't call
__del__ methods on objects *in* trash cycles, because there's no
principled way to know in which order they should be called.  But it's
perfectly happy to invoke __del__ methods on trash objects reachable
only from trash cycles (that are not themselves in a cycle).  Applying
this consistently can make __del__ methods much easier to live with: 
it doesn't matter then how many cycles your "real" objects are
involved in, or how messy they are.  The simple objects with the
__del__ methods are not in cycles then, and so the cycles don't create
any problems for them.

When a weakref and its weak referent are both in cyclic trash, Python
currently says "the order in which they die is undefined, so we'll
pretend the weakref dies first", and then, as at the start of this
msg, the callback is never invoked.  The problem this avoids is that
the callback may itself be part of cyclic trash too, in which case
it's possible for the callback to resurrect anything else in cyclic
trash, including the weakly-referenced object associated with the
callback.  But by hypothesis in this case, that object is also in
cyclic trash, and horrible things can happen if a partially destructed
object gets resurrected.

So the real rule right now is that, if you want a guarantee that a
callback on a weakly referenced object gets invoked, you have to write
your code to guarantee that the referent becomes trash before its
weakref becomes trash.  (And note that if you do, the callback
necessarily will be alive too, because a weakref does in fact hold a
strong reference to its callback.)