[Python-Dev] reference leaks

[Neal Norwitz]

There are still a few reference leaks I've been able to identify.  I
didn't see an obvious solution to these (well, I saw one obvious
solution which crashed, so obviously I was wrong).

When running regrtest with -R here are the ref leaks reported:

test_codeccallbacks leaked [2, 2, 2, 2] references
test_compiler leaked [176, 242, 202, 248] references
test_generators leaked [254, 254, 254, 254] references
test_tcl leaked [35, 35, 35, 35] references
test_threading_local leaked [36, 36, 28, 36] references
test_urllib2 leaked [-130, 70, -120, 60] references

test_compiler and test_urllib2 are probably not real leaks, but data
being cached.  I'm not really sure if test_tcl is a leak or not. 
Since there's a lot that goes on under the covers.  I didn't see
anything obvious in _tkinter.c.

I have no idea about test_threading_local.

I'm pretty certain test_codeccallbacks and test_generators are leaks. 
Here is code that I gleaned/modified from the tests and causes leaks
in the interpreter:

#### test_codeccallbacks

import codecs
def test_callbacks():
  def handler(exc):
    l = [u"<%d>" % ord(exc.object[pos]) for pos in xrange(exc.start, exc.end)]
    return (u"[%s]" % u"".join(l), exc.end)
  codecs.register_error("test.handler", handler)
  # the {} is necessary to cause the leak, {} can hold data too
  codecs.charmap_decode("abc", "test.handler", {})

test_callbacks()
# leak from PyUnicode_DecodeCharmap() each time test_callbacks() is called

#### test_generators

from itertools import tee

def fib():
  def yield_identity_forever(g):
    while 1:
      yield g
  def _fib():
    for i in yield_identity_forever(head):
      yield i
  head, tail, result = tee(_fib(), 3)
  return result

x = fib()
# x.next() leak from itertool.tee()

####

The itertools.tee() fix I thought was quite obvious:

+++ Modules/itertoolsmodule.c   (working copy)
@@ -356,7 +356,8 @@
 {
        if (tdo->nextlink == NULL)
                tdo->nextlink = teedataobject_new(tdo->it);
-       Py_INCREF(tdo->nextlink);
+       else
+               Py_INCREF(tdo->nextlink);
        return tdo->nextlink;
 }

However, this creates problems elsewhere.  I think test_heapq crashed
when I added this fix.  The patch also didn't fix all the leaks, just
a bunch of them.  So clearly there's more going on that I'm not
getting.

n