[pypy-issue] Issue #2444: PyPy/cpyext: Reference counting leak when using the buffer protocol (pypy/pypy)

[Wenzel Jakob]

New issue 2444: PyPy/cpyext: Reference counting leak when using the buffer protocol
https://bitbucket.org/pypy/pypy/issues/2444/pypy-cpyext-reference-counting-leak-when

Wenzel Jakob:

Hi,

I encountered the following issue which became apparent when adding PyPy support to pybind11 and trying to get the test suite to pass.

PyPy currently leaks instances of cpyext types implementing the buffer protocol even when matched calls to ``PyObject_GetBuffer`` and ``PyBuffer_Release`` are used.

To reproduce, do the following:

```
#!bash
$ git clone https://github.com/wjakob/pybind11
$ cd pybind11
$ cat <<EOF > example.cpp 
#include <pybind11/pybind11.h>
#include <iostream>

namespace py = pybind11;

PYBIND11_PLUGIN(example) {
    py::module m("example");

    struct Buffer {
        float f = 42.0;

        Buffer() { std::cout << "Buffer constructed." << std::endl; }
        virtual ~Buffer() { std::cout << "Buffer destructed." << std::endl; }
    };

    py::class_<Buffer>(m, "Buffer", py::buffer_protocol())
       /* Default constructor */
       .def(py::init<>())
       /* Constructor taking an instance satisfying the buffer protocol (buffer will be requested and immediately released) */
       .def("__init__", [](Buffer &buffer, py::buffer b) { new (&buffer) Buffer(); b.request(); })
       /* Callback which provides access to the internal buffer */
       .def_buffer([](Buffer &m) -> py::buffer_info {
            return py::buffer_info(
                &m.f,             /* Pointer to buffer */
                sizeof(float),    /* Size of one scalar */
                "f",              /* Python struct-style format descriptor */
                1,                /* Number of dimensions (scalar) */
                { 1 },            /* Shape */
                { sizeof(float) } /* Strides */
            );
        });

    return m.ptr();
}
EOF
```


Expected behavior (shown here on Python 2.7).
```
#!bash
# Compile module
# (remove the -undefined dynamic_lookup parameter if compiling on linux)
$ g++ example.cpp -o example.so -I include -rdynamic `python2.7-config --cflags` -I include -std=c++11 -shared -undefined dynamic_lookup
# Try it:
$ python2.7
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import example
>>> import numpy as np
>>> np.array(example.Buffer())
Buffer constructed.
Buffer destructed.
array([ 42.], dtype=float32)
```

Observed behavior on PyPy:
```
#!bash
# Compile the module
# (remove the -undefined dynamic_lookup parameter if compiling on linux)
$ g++ example.cpp -o example.pypy-41.so -I include -rdynamic -I <path-to-pypy-include-dir> -I include -std=c++11 -shared -undefined dynamic_lookup
# Try it:
~/pypy/pypy/goal/pypy-c
Python 2.7.12 (98f8c7e783db, Nov 23 2016, 15:31:26)
[PyPy 5.7.0-alpha0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``the new pypy sport is to pass
pypy bugs as cpython bugs''
>>>> import example
>>>> import numpy as np
>>>> np.array(example.Buffer())
Buffer constructed.
array([ 42.], dtype=float32)
>>>> import gc
>>>> gc.collect()
0
>>>> gc.collect()
0
>>>> gc.collect()
0
>>>> # Uh oh, Buffer instance is never getting freed
```

Note that NumPy is not to blame here -- the issue appears to be within PyPy. I added a second constructor in the example, which allows Buffer to be constructed from itself. And again:

```
$ ~/pypy/pypy/goal/pypy-c
Python 2.7.12 (98f8c7e783db, Nov 23 2016, 15:31:26)
[PyPy 5.7.0-alpha0 with GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``NO VCS DISCUSSIONS''
>>>> import example
>>>> example.Buffer(example.Buffer())
Buffer constructed.
Buffer constructed.
<example.Buffer object at 0x000000010dffb050>
>>>> import gc
>>>> gc.collect()
0
>>>> gc.collect()
Buffer destructed.
0
>>>> gc.collect()
0
>>>> # Uh oh, second Buffer instance is never getting freed
```