Exposing buffer interface for non-extension types?

[Carl Banks]

On Jul 20, 6:04 pm, Ken Watford <kwatford+pyt... at gmail.com> wrote:
> On Tue, Jul 20, 2010 at 8:28 PM, Carl Banks <pavlovevide... at gmail.com> wrote:
> > On Jul 20, 3:09 pm, Ken Watford <kwatford+pyt... at gmail.com> wrote:
> >> Is there any way to expose the PEP 3118 buffer interface for objects
> >> that aren't extension types?
>
> >> Currently, I can expose the NumPy array interface (using either
> >> __array_interface__ or __array_struct__) for any class, extension or
> >> otherwise. But I can't find any reference to python-side interfacing
> >> for PEP 3118. SWIG makes an extension module for your wrapped code,
> >> but not extension *types*, so the classes it produces are pure-python
> >> with methods added in from the extension module.
>
> >> The NumPy array interface works fine for now (especially since NumPy
> >> is the only thing I need to consume these objects), but the
> >> documentation claims that it's being deprecated in favor of PEP 3118,
> >> so I thought it might be relevant to bring this up.
>
> > Can you tell let us know what you want to use it for?  We could offer
> > better help.
>
> > Numpy is generally how I get at buffers in Python 2.x.  For instance
> > if I have an object m that supports buffer protocol (m could a string,
> > mmap object, Python array, etc.), then the following will create an
> > array view of the same buffer:
>
> > numpy.ndarray((10,10),type=numpy.float32,buffer=m)
>
> > As far as I know this procedure won't be too different under PEP 3118;
> > if anything it's simplified in Python 3 since it can discover type and
> > shape information itself.  (You'll have to check with the numpy people
> > on that.)
>
> I'm not having trouble using buffers, I'm having trouble providing them.

I see.


> As a part of SWIG-wrapping a larger C++ project, I'm producing some
> wrappings for Blitz++ arrays. I can extract the shape and stride
> information from the array object to fill either NumPy's or PEP 3118's
> appropriate structure. In the case of NumPy, I can easily arrange for
> the necessary interface on the proxy object to be fulfilled, because
> NumPy doesn't care what kind of object it's attached to. But the PEP
> 3118 interface can only be provided by C extension types.

Well, let's get the facts straight before proceeding.  The buffer
protocol has always been for C only.  PEP 3118 is an extension to this
buffer protocol, so it's also C only.  Numpy's array interface is not
the buffer protocol, and is specific to numpy (though I guess other
types are free to use it among themselves, if they wish).  So what
you've been doing was *not* to provide buffers; it was to provide some
numpy-specific methods that allowed numpy to discover your buffers.

That said, your complaint is reasonable.  And since the numpy docs say
"use PEP 3118 instead of array interface" your misunderstanding is
understandable.

I assume it's not practical to address your real problem (i.e., SWIG),
so we'll instead take it for granted that you have a block of memory
in some custom C code that you can't expose as a real buffer.  In the
past, you've used numpy's special Python methods to let numpy know
where your memory block was, but what do you do now?

Your suggestion to write a small base class is probably the best you
can do as things stand now.  Another thing you can do is write a small
extension to create a surrogate buffer object (that implements buffer
protocol), so if you want to create an ndarray from a buffer you can
do it like this:

    b = create_surrogate_buffer(pointer=_proxy.get_buffer_pointer())
 
numpy.ndarray(shape=_proxy.get_shape(),type=_proxy.get_type(),buffer=b)

Not as convenient to use, but might be simpler to code up than the
base-class solution.

The numpy developers might be sympathetic to your concerns since you
have a reasonable use case (they might also tell you you need to get a
real wrapper generator).  Just be sure to keep the distinction clear
between "buffer protocol" (which is what PEP 3118 is, is C only, and
the buffer must be owned by a Python object) and "array
interface" (which is what you were using for 2.x, is C or Python, and
the buffer can be anywhere).

HTH


Carl Banks