[Python-Dev] Avoiding reference leaks in heap types with custom tp_dealloc

[Petr Viktorin]

Hello,
The new test_importlib.extension.test_loader is currently leaking
references, (issue24268). There is a simple hack to stop this, but I'm
inclined to not apply quick hacks and rather dig into the root cause.
(It's a test module, the refleaks are relatively harmless.)

The tests are based directly on the "xxlimited" example,
xxlimited.Xxo, which exhibits the same bug -- it's just not tested.
It's is caused by a combination of a few factors, but I'm not sure
what's a bug and what's just undocumented behavior, so I'm asking for
input to put me on the right track.

As reported in issue16690, heap types with a naïve custom tp_dealloc
leak references to the type when instantiated. According to [0], it
seems that tp_dealloc should check if it has been overridden, and if
so, decref the type. This needs to be documented (regardless of the
solution to the other problems), and I intend to document it.
We can change xxlimited to do the check and decref, but isn't it too
ugly for a module that showcases the extension module API?
(xxlimited.Xxo can technically skip the check, since it doesn't allow
subclasses, but that would be setting a nasty trap for anyone learning
from that example.)

The nice way out would be taking advantage of PEP 442: xxlimited.Xxo
can ditch tp_dealloc in favor of tp_traverse and tp_finalize (the
former of which it needs anyway to behave correctly). Unfortunately,
tp_finalize is not available in the stable ABI (issue24345). I think
it should be added; is it too late for 3.5?

Another problem is that xxlimited is untested. It's only built for
non-debug builds, because Py_LIMITED_API and Py_DEBUG are
incompatible. Would it make sense to build and test it without
Py_LIMITED_API in debug mode, instead of not building it at all?


[0] http://bugs.python.org/issue15653#msg168449