[Cython] Memory leak of memoryview attributes in cdef subclasses

[Daniel, Bruno]

Dear developers,

We encountered memory leaks when using memoryview attributes in cdef subclasses
in Cython code. They can be avoided by adding a dealloc method setting the value
of the memoryview attribute to None. The problem does not occur in topmost
cdef classes.

Here's an example: (See the module docstring on how to compile and run the
example.)


--- File memoryview_leak.pyx --------------------------------------------------
"""
Cython module exposing memory leaks due to memoryview attributes in cdef
classes

Compile::

    python setup.py build_ext --inplace

Run::

    python -c "import memoryview_leak; memoryview_leak.main()"
"""
from __future__ import print_function, division

import unittest
import gc

import numpy as np


cdef class LeakTestSuper:
    """
    Superclass for the leak test in :class:`LeakTest1` and :class:`LeakTest2`.
    """
    pass
   

cdef class LeakTest1(LeakTestSuper):
    """
    Class for memory leak testing.

    Holds a memoryview which is extracted from an :class:`numpy.ndarray`
    at initialization.

    This class avoids the leak by resetting the memoryview to None in
    the method :meth:`__dealloc__`.

    ``__del__()`` wouldn't work, because it is not supported in Cython's
    `cdef` classes.
    """
    cdef double[:] _s
   
    def __init__(self):
        self._s = np.empty(2)

    def __dealloc__(self):
        self._s = None


cdef class LeakTest2(LeakTestSuper):
    """
    Class for memory leak testing.

    Holds an array that is allocated at initialization.
   
    This class does not avoid the leak and thus exposes the problem with
    memoryviews in Cython.
    """
    cdef double[:] _s
   
    def __init__(self):
        self._s = np.empty(2)


class TestLeakTest(unittest.TestCase):
   
    def n_objects_by_type(self, typename):
        return len([
            obj for obj in gc.get_objects()
            if type(obj).__name__ == typename])

    def test_gc(self):
        # Make sure to clear all old memoryviews (circularly referenced).
        gc.collect()
        # Now there mustn't be any memoryviews left.
        n_memory_views_expexted = 0
       
        for cause_leak in [False, True, False, True]:
            # Create an object of LeakTest1 or LeakTest2 allocating a memoryview
            # internally.
            leaktest = LeakTest2() if cause_leak else LeakTest1()

            # Check the number of allocated memory views.
            n_memory_views_expexted += 1
            n_memory_views = self.n_objects_by_type('memoryview')
            self.assertEqual(n_memory_views, n_memory_views_expexted)
           
            # Delete the reference to leaktest and let the garbage collector do
            # its thing.
            del leaktest
            gc.collect()

            # Check for leaks by counting the memoryviews again.
            if not cause_leak:
                n_memory_views_expexted -= 1
            n_memory_views = self.n_objects_by_type('memoryview')
            self.assertEqual(n_memory_views, n_memory_views_expexted)


def main():
    unittest.main()   
-------------------------------------------------------------------------------


This is the corresponding setup file:

--- File setup.py -------------------------------------------------------------
from __future__ import print_function, division
from distutils.core import setup
from Cython.Build import cythonize

setup(
    name = "memoryview_leak",
    ext_modules = cythonize('memoryview_leak.pyx'),
)
-------------------------------------------------------------------------------

If the unit tests are all ok in this example, this means that the memory leak
has been encountered in LeakTest2 and avoided in LeakTest1 as expected.

Best regards,
   Bruno Daniel

-----
Dr. Bruno Daniel
Research and Development
Blue Yonder, Karlsruhe, Germany