Can someone explain this weakref behavior?

[David MacQuigg]

On Fri, 11 Jun 2004 15:38:29 -0400, "Tim Peters" <tim.one at comcast.net>
wrote:

>[Tim Peters]
>>> That will pass under CPython today, but there's no general guarantee
>>> about exactly when a weak dict will notice that keys (or values) have
>>> become unreachable by strong references.
>
>[David MacQuigg]
>> OUCH!!  We just built a module that uses weak references to keep a
>> "robust count" of instances in various classes.  The claim is that this
>> is more robust than simply incrementing and decrementing class variables
>> using __init__ and __del__.   The module seems to be working OK,
>> immediately deleting the weak reference as soon as all references to the
>> corresponding instance are deleted.
>
>Then it must be the case that you're running CPython, and that these
>instances aren't involved in cycles.  Because CPython primarily uses
>reference-counting to recycle garbage, its behavior is predictable in the
>absence of cycles.

I'm not worried about cyclic references, but it is something to keep
in mind.

>CPython's use of reference counting is an implementation detail, and that
>internal weakref lists are traversed "immediately" upon an object's refcount
>reaching 0 is also an implementation detail.  Nothing in the language
>definition guarantees these behaviors.
>
>> If I understand you correctly, there is some chance that a future
>> implementation of Python may have the weak references "out-of-sync" with
>> the actual count of live instances.  Is that a remote possibility, or
>> something quite likely to occur?
>
>Well, it's been the case for a long time in JPython.  I judge the odds of it
>changing in CPython as slim.  I personally wouldn't worry about it ever
>changing in CPython.  If a PyPy- or Parrot-based implementation of Python
>takes off, behavior will depend on its implementation details.
>
>> I have to decide now whether to rip out some risky code.
>>
>> Is there a good way to track the count of instances?
>
>If you want it enough, you can build Python in a mode that tracks this
>automatically (see the discussion of COUNT_ALLOCS in Misc/SpecialBuilds.txt
>-- for each distinct type object, the total # of allocations, # of
>deallocations, and highwater mark (max(#alloc - #dealloc) over time) are
>maintained in a COUNT_ALLOCS build).
>
>> If not, would it make sense to request a guarantee on the current
>> behavior of weak references?  Maybe it could be an option, assuming there
>> is some performance penalty, an option to be used when accuracy is more
>> important than speed.
>
>You can request anything you can dream up <wink>.  If it's something your
>business needs, the only way to guarantee it is to get involved in Python
>development deeply enough so that, if worse comes to worse, you can maintain
>your own Python implementation.  That's unreasonably paranoid in my
>estimation, but it's a judgment call.

Seems like we could do this more easily with a function that lists
instances, like __subclasses__() does with subclasses.  This doesn't
have to be efficient, just reliable.  So when I call
cls.__instances__(), I get a current list of all instances in the
class.

Maybe we could implement this function using weak references.  If I
understand the problem with weak references, we could have a
WeakValueDictionary with references to objects that actually have a
refcount of zero.  There may be too many entries in the dictionary,
but never too few.  In that case, maybe I could just loop over every
item in my WeakValueDictionary, and ignore any with a refcount of
zero.

    def _getInstances(cls):
        d1 = cls.__dict__.get('_instances' , {})
        d2 = {}
        for key in d1:
            if sys.getrefcount(d1[key]) > 0:
                d2[key] = d1[key]
        return d2
    _getInstances = staticmethod(_getInstances)

I'm making some assumptions here that may not be valid, like
sys.getrefcount() for a particular object really will be zero
immediately after all normal references to it are gone. i.e. we don't
have any temporary "out-of-sync" problems like with the weak
references themselves.

Does this seem like a safe strategy?

-- Dave