[Python-Dev] Avoiding reference leaks in heap types with custom tp_dealloc

[Benjamin Peterson]

On Mon, Jun 1, 2015, at 11:33, Antoine Pitrou wrote:
> On Mon, 1 Jun 2015 16:38:35 +0200
> Petr Viktorin <encukou at gmail.com> wrote:
> > Hello,
> > The new test_importlib.extension.test_loader is currently leaking
> > references, (issue24268). There is a simple hack to stop this, but I'm
> > inclined to not apply quick hacks and rather dig into the root cause.
> > (It's a test module, the refleaks are relatively harmless.)
> > 
> > The tests are based directly on the "xxlimited" example,
> > xxlimited.Xxo, which exhibits the same bug -- it's just not tested.
> > It's is caused by a combination of a few factors, but I'm not sure
> > what's a bug and what's just undocumented behavior, so I'm asking for
> > input to put me on the right track.
> 
> Yes, the issue is really nasty. There are several situations to take
> into account:
> - derived heap type inherits from base heap type, both have custom
>   deallocs
> - derived heap type inherits from base heap type, only one has
>   a custom dealloc
> - derived heap type inherits from Python-defined class (the latter
>   having subtype_dealloc)
> - derived heap type inherits from static type
> - ...
> 
> It is unreasonable to expect developers of C extensions come up with
> the correct incantation (and ideally, they shouldn't have to think
> about it at all).
> 
> > The nice way out would be taking advantage of PEP 442: xxlimited.Xxo
> > can ditch tp_dealloc in favor of tp_traverse and tp_finalize (the
> > former of which it needs anyway to behave correctly). Unfortunately,
> > tp_finalize is not available in the stable ABI (issue24345). I think
> > it should be added; is it too late for 3.5?
> 
> Well, but.... the stable ABI is supposed to be a subset of the API
> that's safe to program against, regardless of the Python version (at
> least from the point where the stable ABI was introduced). What happens
> if you define a Py_tp_finalize and run your C extension type on a
> pre-3.5 version? Do you get an error at definition time? A resource
> leak? A crash?
> 
> I don't get why Benjamin committed the change so quick.

I thought all the slots were supposed to be available through the stable
ABI, since you need them to properly define c types.