bugs in `gc.get_referents()'

[Zooko]

[Would this be more appropriate for the Python developers' list?]

Hello Pythonistas, Pythonics and Pythonifiers!

I'm using Python 2.2b2 as compiled and packaged by debian.

I've been using `gc.get_objects()' and `gc.get_referents()' (since renamed 
`gc.get_referrers()') to find out how my app Mojo Nation[1] uses memory.

(Thank you to the implementors of those functions!  It has been a great help!)

There are two apparent bugs: first, the return value from get_referents() 
*always* contains a number of entries that do not refer to the object in 
question, and it is always the same entries.  For example:

>>> t = (0,1,2,3,)
>>> l = [t,]
>>> refs = gc.get_referents(t)
>>> len(refs)
9
>>> hr(refs[0])
'[(0, 1, 2, 3)]'
>>> hr(refs[1])
"{'__builtins__': {'ArithmeticError': <class exceptions.ArithmeticError at 0x8056314>, 'AssertionError': <class exceptions.AssertionError at 0x804dd94>, 'AttributeError': <class exceptions.AttributeError at 0x8054cdc>, 'DeprecationWarning': <class exceptions.DeprecationWarning at 0x80575c4>, 'EOFError': <class exceptions.EOFError at 0x80544b4>, 'Ellipsis': Ellipsis, ...}, 'broker': <__main__.Broker instance at 0x856820c>, 'gc': <module 'gc' (built-in)>, 'hr': <MojoRepr.repr() at repr.py:14>, 'human_readable': <module 'human_readable' from '/home/zooko/playground/evil-SF-branch_stable/common/human_readable.pyo'>, 'l': [(0, 1, 2, 3)], ...}"
>>> hr(refs[2])
"{'Additional_DTML_Paths': [], 'DocumentTemplate': <module 'DocumentTemplate' from '/home/zooko/playground/evil-SF-branch_stable/common/DocumentTemplate/__init__.pyo'>, 'DuplicateChecker': <class Presentation.DuplicateChecker at 0x85161e4>, 'PresentationError': <class Presentation.PresentationError at 0x85161b4>, 'PseudoModule': <class Presentation.PseudoModule at 0x851631c>, 'VariableManager': <class Presentation.VariableManager at 0x851711c>, ...}"
>>> hr(refs[3])
'[<TCPCommsHandler.TCPCommsHandler at 0x8721fa4>, <TCPCommsHandler.TCPCommsHandler listening :23077 at 0x87a07dc>]'
>>> hr(refs[4])
"{'_cid_to_cs': {}, '_mesgen': <mesgen.MessageMaker instance at 0x871ae34>, '_tcpch': <TCPCommsHandler.TCPCommsHandler at 0x8721fa4>, '_upward_inmsg_handler': <ConversationManager.handle_raw_message() at Conversation.py:383>}"
>>> hr(refs[5])
"{'_DELAY': 600, '_attrs': {'_commstratseqno': 0, '_lastannouncedcommstratdict': None}, '_commstratseqno': 0, '_cryptol': <CryptoCommsHandler.CryptoCommsHandler instance at 0x8752844>, '_dirty': None, '_fname': '/home/zooko/playground/.mojonation//broker/mtmdb/P0e0Is12KvUN6HI4aAvwyjJgiVA/TCPListener.pickle', ...}"
>>> hr(refs[6])
'<TCPCommsHandler._readable_false() at TCPCommsHandler.py:451>'
>>> hr(refs[7])
'<TCPCommsHandler._unthrottle() at TCPCommsHandler.py:442>'
>>> hr(refs[8])
'<TCPCommsHandler._throttle() at TCPCommsHandler.py:438>'

The first response is the answer I expected.  The second and third might be some 
local or global scope dict or some other Python guts that I don't understand.  
But the remaining six entries are data structures from my code which do *not* 
refer to the object in question, and curiously those six data structures (or 
ones like them -- I'm not sure) are *always* returned from any call to 
gc.get_referents()!

The second bug is that about once every few hours of use, get_referents() seg 
faults.  I think it might be more likely to do that when I have a larger 
collection of objects in memory -- maybe the time to traverse the larger set of 
objects reveals a race condition?

I'm running gc.get_referents() in an interactive interpreter while several 
threads are simultaneously working and mutating the data!  Also some of those 
threads are using native code which might be failing to DECREF properly.

I would be rather interested in digging into the implementation of 
get_referents() myself, but first I'd like to hear from the Pythonistas whether 
these bugs are surprising or unsurprising, whether the code has been changed 
since 2.2b2, if there is any sort of "starting point" for digging into it that 
they recommend, etc.

Thanks!

Zooko
http://zooko.com/

[1] http://mojonation.net/