[Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7?

[Jim Fulton]

On Fri, Sep 24, 2010 at 3:36 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 24 Sep 2010 15:14:32 -0400
> Tim Peters <tim.peters at gmail.com> wrote:
>> Looks like 2.7 changes introduced to exempt dicts and tuples from
>> cyclic gc if they obviously can't be in cycles has some unintended
>> consequences.  Specifically, if an extension module calls
>> PyObject_GC_UnTrack() on a dict it _does not want tracked_, Python can
>> start tracking the dict again.
>>
>> I assume this is unintended because (a) the docs weren't changed to
>> warn about this; and, (b) it's wrong ;-)
>
> It was indeed unintended. I didn't know people were using
> PyObject_GC_(Un)Track in other places than constructors and destructors.
>
>> There are two main reasons an extension module may have been calling
>> PyObject_GC_UnTrack():
>>
>> 1. For correctness, if the extension is playing games with reference
>> counts Python isn't expecting.
>
> Yikes :)
>
>> 2. For speed, if the extension is creating dicts (or tuples) it knows
>> cannot participate in cycles.
>
> The optimization is now automated in the simple cases (as you've found
> out!).
>
>> This came up when Jim Fulton asked me for advice about assertion
>> failures coming out of cyclic gc in a debug build when running ZODB's
>> tests under 2.7.  Turned out to be due to the "#1 reason" above:  ZODB
>> hand-rolled its own version of weak references long before Python had
>> them, and has a dict mapping strings ("object IDs") to complex objects
>> where the referenced objects' refcounts intentionally do _not_ account
>> for the references due to this dict.
>
> Perhaps ZODB should switch to standard weak references these days?
> (as a bonus, chances are it will be faster)

This is the long term plan.  Switching is not going to be a small
project and not high on the list of priorities.

(Actually, ZODB invented its own weakref mechanism after Python had
weakrefs, but before weakrefs were subclassable. Using standard
weakrefs was deemed too expensive in terms of memory use.)

For the record, I don't consider this a Python bug.  This corner of
ZODB is living on the edge and deserves what it gets. :)  I'm just
happy the fix was ultimately pretty simple.

>> Best if no changes had been needed.  "Better than nothing" if the docs
>> are changed to warn that the effect of calling PyObject_GC_UnTrack()
>> may be undone by Python a nanosecond later ;-)
>
> A doc addition will be enough, hopefully.

Absolutely.

Jim

--
Jim Fulton