[Python-Dev] Debugging opportunity :-)

[Phillip J. Eby]

At 11:51 PM 4/14/2006 +0200, Thomas Wouters wrote:

>On 4/14/06, Phillip J. Eby 
><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
>>At 03:49 PM 4/14/2006 -0400, Tim Peters wrote:
>> >   That eventually
>> >pointed to an off-by-one error in the new PyGen_NeedsFinalizing(),
>> >where reading up trash made it very likely that old generators
>> >containing an active loop would falsely claim they need finalization.
>> >Alas, fix that, and Python segfaults when running test_generators.
>>
>>Even with the close() gimmicks still in place?  If so, it sounds like the
>>safest thing for a2 might be to claim that generators always need
>>finalizing.  :(
>
>No, it sounds like there's a bug in generator cleanup :-) From Tim's 
>description, it isn't obvious to me that the bug is in the hack to clean 
>up simple generator frames. I had "finding generator-cycle leak in 
>test_generators" on my TODO list (as I'd noticed it was still leaking, 
>too), I've now replaced that with "finding the generator-cleanup crash Tim 
>encountered" :-) I don't know if I'll get to it this weekend, though, so 
>if anyone else wants to hunt for it, don't let me stop you.

FYI, the smallest code I've found so far that reproduces the crash is:

fun_tests = """
 >>> class LazyList:
...     def __init__(self, g):
...         self.v = None
...         self.g = g
...
...     def __iter__(self):
...         yield 1
...         if self.v is None:
...             self.v = self.g.next()
...         yield self.v

 >>> def loop():
...     for i in ll:
...         yield i

 >>> ll = LazyList(loop())
 >>> g=iter(ll)
 >>> g.next()
1
 >>> g.next()
1
"""

If you replace the fun_tests in test_generators with this, and remove all 
the other tests (e.g. pep_tests, tut_tests, etc.) you can still get it to 
crash.  The code above is much shorter, but has the disadvantage that I 
don't really know what it's supposed to do, other than cause a crash.  :)

Interestingly, this code does *not* crash the interpreter on its own, only 
when run as a doctest (with or without regrtest).  I still haven't figured 
out what the actual problem is, but at least this cuts out all the merge() 
and times() crud in the case this was derived from, reducing it to a pure 
zen essence of non-meaning.  :)