[Cython] Memory leak of memoryview attributes in cdef subclasses

[Daniel, Bruno]

Dear Paul,

> Cannot reproduce the failure on Cython 0.19.1. If you are using an
> earlier version, please try that instead.

We also used Cython 0.19.1. If the unit test runs through, this means that
the memory leak is there. I'm sorry for the ambiguity. Please change 
the lines

            if not cause_leak:
                n_memory_views_expexted -= 1

into just

            n_memory_views_expexted -= 1

Then the test will fail if the memory leak exists.

I also just noticed that    

     unittest.main()    

doesn't work in Cython files. The tests are just ignored. Please call the
tests this way instead:  

python -m unittest memoryview_leak

For your convenience, here's the updated code:

-------- File memoryview_leak.pyx ---------------------------------------------------------------
"""
Cython module exposing memory leaks due to memoryview attributes in cdef
classes in Cython 0.19.1

Compile::

    python setup.py build_ext --inplace

Run::

    python -m unittest memoryview_leak
"""
from __future__ import print_function, division

import unittest
import gc

import numpy as np


cdef class LeakTestSuper:
    """
    Superclass for the leak test in :class:`LeakTest1` and :class:`LeakTest2`.
    """
    pass
    

cdef class LeakTest1(LeakTestSuper):
    """
    Class for memory leak testing.

    Holds a memoryview which is extracted from an :class:`numpy.ndarray`
    at initialization.

    This class avoids the leak by resetting the memoryview to None in
    the method :meth:`__dealloc__`.

    ``__del__()`` wouldn't work, because it is not supported in Cython's
    `cdef` classes.
    """
    cdef double[:] _s
    
    def __init__(self):
        self._s = np.empty(2)

    def __dealloc__(self):
        self._s = None


cdef class LeakTest2(LeakTestSuper):
    """
    Class for memory leak testing.

    Holds an array that is allocated at initialization.
    
    This class does not avoid the leak and thus exposes the problem with
    memoryviews in Cython.
    """
    cdef double[:] _s
    
    def __init__(self):
        self._s = np.empty(2)


class TestLeakTest(unittest.TestCase):
    
    def n_objects_by_type(self, typename):
        return len([
            obj for obj in gc.get_objects()
            if type(obj).__name__ == typename])

    def test_gc(self):
        # Make sure to clear all old memoryviews (circularly referenced).
        gc.collect()
        # Now there mustn't be any memoryviews left.
        n_memory_views_expexted = 0
        
        for cause_leak in [False, True, False, True]:
            # Create an object of LeakTest1 or LeakTest2 allocating a memoryview
            # internally.
            leaktest = LeakTest2() if cause_leak else LeakTest1()

            # Check the number of allocated memory views.
            n_memory_views_expexted += 1
            n_memory_views = self.n_objects_by_type('memoryview')
            self.assertEqual(n_memory_views, n_memory_views_expexted)
            
            # Delete the reference to leaktest and let the garbage collector do
            # its thing.
            del leaktest
            gc.collect()

            # Check for leaks by counting the memoryviews again.
            if True: # not cause_leak:
                n_memory_views_expexted -= 1
            n_memory_views = self.n_objects_by_type('memoryview')
            self.assertEqual(n_memory_views, n_memory_views_expexted)
-----------------------------------------------------------------------------------------------------

Here's the corresponding setup file:
-------- File setup.py ----------------------------------------------------------------------------
from __future__ import print_function, division
from distutils.core import setup
from Cython.Build import cythonize

setup(
    name = "memoryview_leak",
    ext_modules = cythonize('memoryview_leak.pyx'),
)
---------------------------------------------------------------------------------------------------------

Best regards,
    Bruno Daniel


From: cython-devel [cython-devel-bounces+bruno.daniel=blue-yonder.com at python.org] on behalf of cython-devel-request at python.org [cython-devel-request at python.org]
Sent: Wednesday, September 25, 2013 12:00 PM
To: cython-devel at python.org
Subject: cython-devel Digest, Vol 32, Issue 10

Send cython-devel mailing list submissions to
        cython-devel at python.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.python.org/mailman/listinfo/cython-devel
or, via email, send a message with subject or body 'help' to
        cython-devel-request at python.org

You can reach the person managing the list at
        cython-devel-owner at python.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of cython-devel digest..."


Today's Topics:

   1. Re: Memory leak of memoryview attributes in cdef subclasses
      (Pauli Virtanen)


----------------------------------------------------------------------

Message: 1
Date: Tue, 24 Sep 2013 20:43:14 +0300
From: Pauli Virtanen <pav at iki.fi>
To: cython-devel at python.org
Subject: Re: [Cython] Memory leak of memoryview attributes in cdef
        subclasses
Message-ID: <l1siva$rg5$1 at ger.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1

Hi,

18.09.2013 16:14, Daniel, Bruno kirjoitti:
> We encountered memory leaks when using memoryview attributes in cdef subclasses
> in Cython code. They can be avoided by adding a dealloc method setting the value
> of the memoryview attribute to None. The problem does not occur in topmost
> cdef classes.
>
> Here's an example: (See the module docstring on how to compile and run the
> example.)

Cannot reproduce the failure on Cython 0.19.1. If you are using an
earlier version, please try that instead.

One bug with memory leaks in cdef classes was fixed in 0.18 IIRC.

--
Pauli Virtanen



------------------------------

Subject: Digest Footer

_______________________________________________
cython-devel mailing list
cython-devel at python.org
https://mail.python.org/mailman/listinfo/cython-devel


------------------------------

End of cython-devel Digest, Vol 32, Issue 10
********************************************