[Python-Dev] Debugging opportunity :-)

[Thomas Wouters]

On 4/15/06, Phillip J. Eby <pje at telecommunity.com> wrote:

> Interestingly, this code does *not* crash the interpreter on its own, only
> when run as a doctest (with or without regrtest).


Actually, it does, you just have to force GC to clean up stuff (rather than
wait for the interpreter-exit to just toss the whole lot out without giving
it a second glance.) You do that by making it unreachable and calling
gc.collect().

centurion:~/python/python/trunk > cat test_gencrash.py
import gc

class LazyList:
    def __init__(self, g):
        self.v = None
        self.g = g

    def __iter__(self):
        yield 1
        if self.v is None:
            self.v = self.g.next()
        yield self.v

def loop():
    for i in ll:
        yield i

ll = LazyList(loop())
g=iter(ll)
g.next()

g.next()

del g, ll
print gc.collect()

centurion:~/python/python/trunk > ./python test_gencrash.py
Segmentation fault (core dumped)



I still haven't figured
> out what the actual problem is, but at least this cuts out all the merge()
> and times() crud in the case this was derived from, reducing it to a pure
> zen essence of non-meaning.  :)


I'm sure I know where the crash is coming from: one of the generator objects
is being dealloced twice. The GC module tp_clears one of the frame objects,
which deallocs a generator the frame referenced. That first dealloc calls
_Py_ForgetReference and then calls gen_dealloc, which calls its tp_del (as
it's a running generator, albeit a simple one), which raises an exception in
the frame, which causes it to be cleaned up, which causes another generator
object to be cleaned up and its tp_del method to be called, which - for some
reason - tries to dealloc the first generator again. The second dealloc
calls _Py_ForgetReference again, and _Py_ForgetReference doesn't like that
(the ob_prev/ob_next debug ptrs are already wiped when it's called the
second time, so it crashes.)

The thing I don't understand is how the cleaning up of the second generator
triggers a dealloc of the first generator. The refcount is 0 when the
dealloc is called the first time (or _Py_ForgetReference, as well as a few
other places, would have bombed out), and it is also 0 when it gets
dealloced the second time. Since deallocation is triggered by DECREF'ing,
that implies something is INCREF'ing the first generator somewhere -- but
without having a valid reference, since the generator has been cleaned up by
actually DECREF'ing the reference that a frame object held. Or, somehow,
deallocation is triggered by something other than a DECREF. Hmm. No, the
second dealloc is triggered by the Py_XDECREF in ceval.c, when clearing the
stack after an exception, which is the right thing to do. Time to set some
breakpoints and step through the code, I guess.

(I first thought the problem was caused by gen_dealloc doing
'Py_DECREF(gen->gen_frame)' instead of 'frame = gen->gen_frame;
gen->gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It should
do it that way, I believe, but it's not the cause of this crash.)

--
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060415/04d86a94/attachment.htm