[Python-Dev] Cycle collection enhancement idea

[Eyal Lotem]

On Sun, Jun 29, 2008 at 9:00 PM, "Martin v. Löwis" <martin at v.loewis.de> wrote:
>> As I explained above, it *is* part of a cycle: """including
>> the class objects themselves: class->dict->function->func_globals""".
>
> Ah, right. I must have missed that explanation.
>
>> I know. I assumed Python does not rely on cyclic garbage collectioin
>> for shutdown, because it wouldn't work, as _all_ globals that have
>> *any* instance method will be part of a cycle, and any of them which
>> have a __del__ will not be collected.
>
> No. The mechanism for cleaning modules at shutdown time predates cyclic
> GC, and was not removed because "it wouldn't work".
>
> This specific issue certainly contributes to the fact that it doesn't
> work, but there might be other problems as well (such as extension
> modules holding onto objects participating in cycles).
>
>> I *mentioned* this workaround. What I propose is not a workaround but
>> a solution. You wouldn't need to clean up module globals ad-hoc'ishly,
>> because the cyclic collection would collect your object, even with its
>> __del__.
>
> I don't think it solves the problem. You seem to be assuming that any
> such cycle will contain only a single global object with an __del__.
> However, as modules refer to each other, I very much doubt that is the
> case.
>
>> Please read my entire mail before replying to it. Thanks!
>
> I really, really tried. I read it three times before replying.
>
> However, I found it really, really difficult to follow your writing,
> as it was mixing problem statement and solution, so that I couldn't
> tell what paragraph was about what. English is not my native language,
> complicating communication further. Please accept my apologies.

I apologize for my tone as well, it just seemed frustrating that all
the replies seemed to completely ignore the core point I was trying to
make and claim that the problem I was trying to solve was not a
problem at all, but a behavior I was unaware of...

Mixing another quote from another mail:
> > That would be no worse than what happens now - but its still not
> > perfect (__del__ ordering issues). Also, you would need to temporarily
> > revive the cycles as mentioned above (to avoid accessibility of
> > partially destructed objects).
>
> I don't quite understand what you mean by "revive cycles". There is
> not need to revive any object in a cycle, as all objects are still alive
> (or else they wouldn't be garbage).
By "revive cycles", I mean make sure that they are referenced by an
independent referrer (one that won't go away as part of the __del__
calling process).  This is similar to how the tp_dealloc code
increases the refcount (actually sets it to 1, because it was
certainly 0 when entering the destructor) before calling the __del__
slot.  Without reviving the object before calling its __del__ in the
destructor, and without reviving the objects of a cycle before calling
its __del__'s, the __del__ Pythonic code may be exposed to "dead
objects" (refcount==0).

Consider the cycle:
a.x = b
b.x = a

Lets suppose the a object has a __del__. Lets assume each object in
the cycle has a refcount of 1 (and the cycle should die). Now lets say
this is a's __del__ code:
def __del__(self):
  self.x = None

Running it will set 'b's refcount to 0 and call its destructor, which
will set 'a's refcount to 0 and also call its destructor. But its
__del__ is currently running - so "self" must not have a refcount of
0.  If you only incref on 'a' before calling __del__, then you are
probably alright, as long as there is only one __del__.

> Can you please elaborate? What would such __del__ ordering issues be?

class A(object):
  def __del__(self):
    print self.x.attribute

class B(object):
  def __del__(self):
    print "B is going down!"
    del self.attribute

a = A()
b = B()
a.x = b
b.attribute = 1

If you call b's __del__ first then a's __del__ will fail. If you call
a's __del__ first, then all is well.  Ofcourse you can create true
cyclic dependencies that no order will work, and its pretty clear
there is no way to deduce the right order anyway.  This is what I mean
by "ordering issues".

> There could be a global barricade for calling __del__: you first call
> all __del__s of existing objects, then set the barricade, and then
> start breaking cycles.
> This could even be done with the current approach to module clearing.

Note that the __del__'s themselves may be breaking cycles and
refcounts will go to 0 - unless you temporarily revive (incref) the
entire cycle first.

> I still don't understand what "revive the cycle" means. You will need to
> incref the object for which you call __del__, that's all.

Unless there are multiple __del__'s in the cycle.

>
> Regards,
> Martin
>