painful debugging: techniques?

[Peter Otten]

Humpty Dumpty wrote:

> Hello, I've been using python for about 3 months now. Prior to that I did
> C++ for over 5 years. I'm wondering if my debugging techniques are too C++
> oriented.
> 
> E.g., it took me about 9 hrs of debugging to figure out that the second
> parameter to weakref.ref() was being passed as None. This is odd because
> the second parameter is optional, yet its default value is not documented,
> so I assumed that passing None would be fine, and then I forgot that I was
> doing that. It took that long mostly because the exception message was
> "Exception TypeError: 'NoneType not a callable' in None being ignored", ie
> it was being ignored and function from which it was being raised was
> unknown to Python (somehow), *and* it was only being raised upon exit of
> the test (cleanup). So I had no way of knowing where this was being
> raised. It was nasty.
> 
> I'm wondering if there are tricks people use to track down problems like
> this.

The best strategy would have been to avoid the error altogether :-)
Python makes that easy; just test a chunk of code on the command line:

>>> import weakref
>>> class X:
...     pass
...
>>> def cb(a): print "callback"
...
>>> x = X()
>>> w = weakref.ref(x, cb)
>>> del x
callback
>>> x = X()
>>> w = weakref.ref(x, None)
>>> del x
Exception exceptions.TypeError: "'NoneType' object is not callable" in None
ignored

The next best thing is to have a proper test suite and making only small
changes to your program, so that you can easily attribute an error to a
newly added chunk of code. IMHO this is the most promising aproach for
anything comprising more than one module. I admit that I don't know how to
devise test to discover your problem.

So what if all else fails? Let's stick to your example. 
I would see not getting a traceback as a strong hint that something unusual
is going on. In this case you can leverage the fact that python is open
source and search for the place where the error message is generated.
Looking for "%r ignored" and "%s ignored" in *.py of Python's source
distribution yields no result. At this point I would extend the search to
"ignored" in the C source: 48 hits, but most of them in comments. The one
in error.c seems promising. Unfortunately the comment in
PyErr_WriteUnraisable() gives the "wrong" __del__() example, so you need
one more iteration to hunt for PyErr_WriteUnraisable. As the variable names
are well chosen, the match in weakrefobject.c
(PyErr_WriteUnraisable(callback);) gives you a strong clue.

I admit this is tedious and that I had a headstart as you already provided
the connection to weakref, but I reckon that this approach would still take
less than one hour. As a hidden benefit, you learn something about Python's
architecture. Also, in my experience your problem is a particularly hard
one to track down (as far as pure Python is concerned).

At last, what I would call the Columbus approach: do you really need
weakrefs in your application, or is the error an indication that you are
introducing unnecessary complexity in your app? Few eggs actually need to
stand on their tip :-)

Peter