[Cython] Python C-api ref count semantics

[Stefan Behnel]

Chris Colbert, 18.02.2011 03:23:
> What is the rule of thumb when declaring functions from python's C-api when
> comes to ref counting?

The general rule is to not declare them yourself. Instead, cimport them 
from the cpython package. (See Cython/Includes/)


> If I define a test case like so:
>
> cdef extern from "Python.h":
>
>      object PyWeakref_NewRef(object, object)
>      object PyWeakref_GET_OBJECT(object)
>
>
> class Foo(object):
>      pass
>
>
> cdef class Test:
>
>      cdef object obj
>      cdef object wr
>
>      def __init__(self):
>          self.obj = Foo()
>          self.wr = PyWeakref_NewRef(self.obj, None)
>
>      def get_ref(self):
>          return PyWeakref_GET_OBJECT(self.wr)
>
>
> I get these random python fatal errors:
>
> In [8]: %timeit -n 1000 t.get_ref()
> 1000 loops, best of 3: 224 ns per loop
>
> In [9]: %timeit -n 1000 t.get_ref()
> Fatal Python error: deallocating None
> Abort trap
>
>
> However, if I redefine the macro signature and getter function to this:
>
> from cpython cimport PyObject
>
> cdef extern from "Python.h":
>
>      object PyWeakref_NewRef(object, object)
>      PyObject* PyWeakref_GET_OBJECT(object)
>
>
> class Foo(object):
>      pass
>
>
> cdef class Test:
>
>      cdef object obj
>      cdef object wr
>
>      def __init__(self):
>          self.obj = Foo()
>          self.wr = PyWeakref_NewRef(self.obj, None)
>
>      def clear_obj(self):
>          self.obj = None
>
>      def get_ref(self):
>          return<object>PyWeakref_GET_OBJECT(self.wr)
>
>
> Then it runs without issue. I can other gather is has to due the
> incref/decref going on in the generated C code. Should be doing something on
> my end to manually manage ref counts when using the C-Api?

Check the CPython documentation. Whenever a function returns a borrowed 
reference, you must declare it as PyObject* and cast it to <object>.

That being said, support for borrowed references has been long on the list 
but no-one has shown interest in doing it (or getting it done) so far.

Stefan