bugs in `gc.get_referents()'

[Zooko]

[Please Cc: zooko at zooko.com in follow-ups.  Thanks!]

Many thanks to Martin von Loewis and Michael Hudson for their prompt and 
informative responses.

One thing I've figured out is that `gc.get_refer{ent,rer}s()' is using equality 
rather than identity testing.  I've opened a bug report on this: [1]

That does not by itself explain the "wrong referrers reported" behaviour.

Martin von Loewis suggested that "there is a bug in the C type of the 
tp_traverse function of [the bogus referrer] object", but this example shows that 
this must not be the explanation:

>>> refs = gc.get_referents([1,[2.3,4,],5,6,[7,8,],[9,]])
>>> len(refs)
8
>>> refs[1]
[<TCPCommsHandler.TCPCommsHandler at 0x85a474c>, <TCPCommsHandler.TCPCommsHandler listening :4052 at 0x85cdadc>]

(I'm pretty sure there is no list anywhere in memory that equals that argument 
to `get_referents()'.)

So the bogus referrer object in this case is a list -- a builtin Python list.  
That particular list (containing those two particular objects of mine) shows up 
in about half of the bogus referrer lists I see.

And by the way, the following two lines consistently segfault when used 
interactively in my running Mojo Nation app (although sometimes it takes a while 
to segfault):

>>> for i in xrange(2**20):
>>>  refs = gc.get_referents({})

I wasn't able to get the same results with any test case simpler than a full 
running Mojo Nation app.  If you want to try it in a full running Mojo Nation 
node then execute the "cvsinstall.txt" script[2] (which fetches the latest Mojo 
Nation source from CVS and builds it and launches the Mojo Nation node), but 
replace the call to "Broker" with a call to "Broker --interact" to enter an 
interactive Python interpreter where you can run the tests quoted above.


I, Zooko, wrote the lines prepended with "> > > ":
Michael Hudson wrote the lines prepended with "> > ":
Martin von Loewis wrote the lines prepended with "> ":

> > > Also some of those threads are using native code which might be
> > > failing to DECREF properly.
> >
> > If that's the case, then you've got problems all your own, no?
> 
> That is my suspicion. get_referrers accesses every object that claims
> to be a container, without attempting a garbage collection. Some
> extension types may not expect that, but they would be broken already.

Waiatasecond: if my native code forgets to DECREF, this can't possibly screw up 
the gcmodule!  There's absolutely no different between my native code forgetting 
to DECREF and my native code actually keeping a live reference (perhaps the only 
live reference) to an object.

On the other hand, if my native code over-DECREFs (or under-INCREFs, etc.), then 
this could surely cause the kind of wackiness quoted above.  If that's the 
problem then I'm surprised that I'm not getting segfaults in normal usage (i.e., 
I've never gotten a segfault except the one described above) nor any *other* 
unusual behavior in normal usage that I've noticed.


Okay, so I'll go through all the native code that we use checking for correct 
reference counting.  I've already seen some failure-to-DECREF in some of our 
native code (but code that is disabled for all the tests that I've described 
here) so I won't be surprised to find other refcount errors.

By the way I had a look at the implementation of `get_referrers()' and I agree 
that it is simple and looks right.

Regards,

Zooko

[1] http://sourceforge.net/tracker/index.php?func=detail&aid=485781&group_id=5470&atid=105470
[2] http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/mojonation/evil/hackerdocs/cvsinstall.txt?rev=HEAD&content-type=text/vnd.viewcvs-markup