[Numpy-discussion] Another reference count leak: ticket #848

[Michael Abbott]

On Tue, 8 Jul 2008, Travis E. Oliphant wrote:
> Michael Abbott wrote:
> > The attached patch fixes another reference count leak in the use of 
> > PyArray_DescrFromType.
> >   
> The first part of this patch is good.  The second is not needed.   
I don't see that.  The second part of the patch addresses the case of an 
early return: this means that the DECREF that occurs later on in the code 
is bypassed, and so a reference leak will still occur if this early return 
case occurs.  Don't forget that PyArray_DescrFromType returns an 
incremented reference that has to be decremented, returned or explicitly 
assigned -- the DECREF obligation has to be met somewhere.

> Also, it would be useful if you could write a test case that shows what 
> is leaking and how you determined that it is leaking.

Roughly

	r = range(n)
	i = 0
	refs = 0
	refs = sys.gettotalrefcount()
	for i in r: float32()
	print refs - sys.gettotalrefcount()

in debug mode python.  This isn't quite the whole story (reference counts 
can be annoyingly fluid), but that's the most of it.  In trunk this leaks 
2 refs per n, with the attached patch there remains one leak I haven't 
chased down yet.

Is there a framework for writing test cases?  I'm constructing tests just 
to pin down leaks that I find in my application (uses numpy and ctypes 
extensively), so they're terribly ad-hoc at the moment.

I'm chasing down leaks in a leaky application (it ate 1G over a weekend), 
and numpy is just one source of leaks -- I suspec that the two reference 
count leaks I've identified so far don't actually leak significant memory, 
so alas my search continues...

> > Definitely not enjoying this low level code.
> What doesn't kill you makes you stronger :-)  
Heh.

> But, you are correct that reference counting is a bear.
Actually, it's not the reference counting I'm grumbling about -- the rules 
for reference counting are, on the whole, pretty straightforward.