[issue38006] Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit

[Tim Peters]

Tim Peters <tim at python.org> added the comment:

> Note that my flags show that W *is* in 'unreachable'.  It has
> to be otherwise F would not have tp_clear called on it.

Right!  W holds a strong reference to F, so if W were thought to be reachable, F would be too.  But F isn't.


> But when delete_garbage() gets done, it moves the object to 'old'.  I
> think that means gc.get_objects() called after the collection completes
> can reveal objects that have had their tp_clear called (if tp_clear
> didn't result in them being freed).

If the tp_clear implementations aren't complete enough to break all cycles, that could happen.

In a world that took cyclic trash "very seriously", it would move them to a new internal list instead, then at the end of the function die if that list isn't empty ("we ran all tp_clears but cyclic trash still remains:  your object implementations inherently leak memory").


> I think the difference is that non-weakref finalizers have strong
> references to objects that they can access when they run. 
> So, if we haven't found them, they will keep all the objects that
> they refer to alive as well (subtract_refs() cannot account for
> those refs).  So those objects will all be valid.

That's persuasive :-)  For essentially the same reason, if a "surprise" finalizer runs during delete_garbage, it can't ressurect (or in any other way access) anything we knew was trash.


> There seems a hole though.  Non-weakref finalizers could have a
> weakref (without callback) to something in the garbage set.  Then,
> when the finalizer runs during delete_garbage(), that finalizer
> code can see non-valid objects via the weakref.  I think this can
> only happen if there are missing/incomplete tp_traverse methods.

Which is what I mean by a "surprise" finalizer:  a trash object T with a finalizer but we don't KNOW T is trash.  If we know T is trash then we force-run its finalizer before delete_garbage starts, so it can only see valid objects.


> We can have finalizer code running during delete_garbage().

Except that was never the intent (quite the contrary!).  The more people have moved to demanding that gc be a full-featured all-purpose collector, the more important it becomes that types implement what such a thing needs.  At a bare minimum, any robust gc needs to be 100% sure of the difference between trash and non-trash, and tp_traverse is the one & only way we have to discover anything about the object graph.  "Almost all" types that can participate in cycles need to implement it.

Or suffer rare shutdown segfaults that somehow nobody ever managed to stumble into before ;-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38006>
_______________________________________