Porting c extension - PyBuffer_New() deprecated in python3. What's the replacement?

[Mark Heieis]

Stefan,

Thank-you for the reply. I hadn't considered cpython, unfortunately the 
extension is too large a project to port at the moment. I ended up 
replacing the PyBuffer_New() segment with malloc() and passing back an 
object from PyByteArray_FromStringAndSize(). It seems to work.

mrh.

On 2014-01-11 01:10, Stefan Behnel wrote:
> Mark Heieis, 11.01.2014 06:47:
>> I need to convert the following existing c extension code to support Python 3.
>>
>> // --- existing code ------
>>
>>      // PyBuffer_New() deprecated in python3
>>      if (!(pyBuf = PyBuffer_New(len)))
>>      {
>>          return NULL;
>>      }
>>
>>      // should use memoryview object in python3
>>      if (PyObject_AsWriteBuffer(pyBuf, &cbuf, &len))
>>      {
>>          Py_DECREF(pyBuf);
>>          return NULL ;
>>      }
>>
>> // fill in cbuf
>>      ...
>>
>>      return pyBuf ;
>>
>> //-----------
>>
>> I'm somewhat confounded in finding an equivalent (PyBuffer_New()) for
>> creating a buffer of size len that has continuous memory in the c extension
>> function for python3. cbuf is manipulated/filled in using c, after which
>> the created pyBuf is then returned. So far, I haven't found much in the way
>> of examples/doc for porting the deprecated Python-/C-level buffer API calls
>> to the new C-level buffer API/memoryview object model.
>>
>> Any guidance or direction to existing doc/example is much appreciated.
> If the extension isn't huge, you should consider rewriting it in Cython.
> That can usually be done quite quickly - the main thing is to figure out
> what the verbose C code actually does and write it down in much simpler
> Python code. And it will make it easy to make the code portable and fast.
> Also likely safer and more generic and versatile, because Cython covers
> away a lot of the annoying boilerplate, ref-counting issues, type
> conversions, etc.
>
> For your specific problem at hand, you could use Cython's memory views:
>
> http://docs.cython.org/src/userguide/memoryviews.html
>
> They allow you to convert the input value to a 1-dim char buffer (or
> whatever you need, but you mentioned the old Py2 buffer interface, which
> can't do much more) by saying
>
>      cdef char[:] my_memview = some_python_object
>
> If you need to pass the unpacked buffer into C code, you can get the
> address as "&my_memview[0]" (i.e. the address of the first item in the
> buffer). Memory views themselves support fast slicing and indexing, so you
> can efficiently work with them using the normal Python slicing/indexing syntax.
>
> In case what you actually receive are not arbitrary buffers but simple byte
> strings or bytearray instances, you can even use the normal byte string
> coercion in Cython and simply say
>
>      cdef char* c_string = some_python_byte_string_object
>
> and then use that pointer to pass it on into C.
>
> I've written a string processing tutorial for Cython here:
>
> http://docs.cython.org/src/tutorial/strings.html
>
> These things may take a moment to learn, especially if you are used to
> doing everything in excessive manual detail in C code, but once you are
> through that, you should get things done much more quickly than when trying
> to do them by hand.
>
> Stefan
>
>