Porting c extension - PyBuffer_New() deprecated in python3. What's the replacement?

[Stefan Behnel]

Mark Heieis, 11.01.2014 06:47:
> I need to convert the following existing c extension code to support Python 3.
> 
> // --- existing code ------
> 
>     // PyBuffer_New() deprecated in python3
>     if (!(pyBuf = PyBuffer_New(len)))
>     {
>         return NULL;
>     }
> 
>     // should use memoryview object in python3
>     if (PyObject_AsWriteBuffer(pyBuf, &cbuf, &len))
>     {
>         Py_DECREF(pyBuf);
>         return NULL ;
>     }
> 
> // fill in cbuf
>     ...
> 
>     return pyBuf ;
> 
> //-----------
> 
> I'm somewhat confounded in finding an equivalent (PyBuffer_New()) for
> creating a buffer of size len that has continuous memory in the c extension
> function for python3. cbuf is manipulated/filled in using c, after which
> the created pyBuf is then returned. So far, I haven't found much in the way
> of examples/doc for porting the deprecated Python-/C-level buffer API calls
> to the new C-level buffer API/memoryview object model.
> 
> Any guidance or direction to existing doc/example is much appreciated.

If the extension isn't huge, you should consider rewriting it in Cython.
That can usually be done quite quickly - the main thing is to figure out
what the verbose C code actually does and write it down in much simpler
Python code. And it will make it easy to make the code portable and fast.
Also likely safer and more generic and versatile, because Cython covers
away a lot of the annoying boilerplate, ref-counting issues, type
conversions, etc.

For your specific problem at hand, you could use Cython's memory views:

http://docs.cython.org/src/userguide/memoryviews.html

They allow you to convert the input value to a 1-dim char buffer (or
whatever you need, but you mentioned the old Py2 buffer interface, which
can't do much more) by saying

    cdef char[:] my_memview = some_python_object

If you need to pass the unpacked buffer into C code, you can get the
address as "&my_memview[0]" (i.e. the address of the first item in the
buffer). Memory views themselves support fast slicing and indexing, so you
can efficiently work with them using the normal Python slicing/indexing syntax.

In case what you actually receive are not arbitrary buffers but simple byte
strings or bytearray instances, you can even use the normal byte string
coercion in Cython and simply say

    cdef char* c_string = some_python_byte_string_object

and then use that pointer to pass it on into C.

I've written a string processing tutorial for Cython here:

http://docs.cython.org/src/tutorial/strings.html

These things may take a moment to learn, especially if you are used to
doing everything in excessive manual detail in C code, but once you are
through that, you should get things done much more quickly than when trying
to do them by hand.

Stefan