[Python-Dev] [Python-checkins] r43358 - python/trunk/Modules/itertoolsmodule.c

[Thomas Wouters]

On 3/27/06, Neal Norwitz <nnorwitz at gmail.com> wrote:
>
> On 3/27/06, Thomas Wouters <thomas at python.org> wrote:
> >
> > The teeobject has GC (hence the word 'and' in 'itertools.tee and its
> > internal teedataobject' ;-) The problem with test_generators is that
> this
> > also leaks:
> >
> > def leak():
> >     def gen():
> >         while True:
> >             yield g
> >     g = gen()
> >
> > leak()
>
> Please add a test for this to the leakers and remove the tee test in
> leakers if that no longer leaks.


I added this test, but I didn't remove test_tee, because it still leaks. I
also checked in a test of code that used to leak, but doesn't anymore. The
cycle this nested generator creates, which is also involved in the test_tee
leak, is not cleanable by the cycle-gc, and it looks like it hasn't been
since the yield-expr/coroutine patch was included in the trunk. I believe
the culprit to be this part of that patch:

Index: Modules/gcmodule.c
===================================================================
--- Modules/gcmodule.c  (revision 39238)
+++ Modules/gcmodule.c  (revision 39239)
@@ -413,10 +413,8 @@
                assert(delstr != NULL);
                return _PyInstance_Lookup(op, delstr) != NULL;
        }
-       else if (PyType_HasFeature(op->ob_type, Py_TPFLAGS_HEAPTYPE))
+       else
                return op->ob_type->tp_del != NULL;
-       else
-               return 0;
 }

 /* Move the objects in unreachable with __del__ methods into `finalizers`.

Now, reverting that part of the patch in revision 39239 triggers an
assertion failure, but removing the assertion makes it work right; the above
nested-generator cycle gets cleaned up again. Later in the trunk, the
assertion was changed anyway, and it no longer fails if I just revert the
gcmodule change. Of course, the reason the cycle is uncleanable is because
generators grew a tp_del method, and that throws cycle-gc in a hissy fit;
reverting the above patch just makes cycle-gc ignore the tp_del of
heaptypes. I don't know enough about the cycle-gc to say whether that's good
or bad, but not having generators be cleanable is not very good. For what
it's worth, reverting the gcmodule patch doesn't seem to break any tests.

However, fixing _both_ those things (itertools.tee lack of GC, and GC's
inability to clean generators) *still does not fix the 254 refleaks in
test_generators*. When I backport the itertools.tee-GC changes to r39238
(one checkin before coroutines), test_generators doesn't leak, neither the
r39238 version of test_generators, nor the trunk version. One revision
later, r39239, either test_generators leaks 15 references. 'Fixing'
gcmodule, which fixes the nested-generator leak, does not change the number
of leaks. And, as you all may be aware of, in the trunk, test_generators
leaks 254 references; this is also the case with the gcmodule fix. So
there's more huntin' afoot.

In the mean time, if people knowledgeable about the cycle-GC care to comment
about the gcmodule change wrt. heaptypes, please do.

--
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060328/c31c1407/attachment.htm