[Python-Dev] PEP 298 - the Fixed Buffer Interface
Guido van Rossum
guido@python.org
Tue, 30 Jul 2002 15:46:41 -0400
> Here is PEP 298 - the Fixed Buffer Interface, posted to
> get feedback from the Python community.
> Enjoy!
+1 from me (but you already knew that).
> Thomas
>
> PS: I'll going to a 2 weeks vacation at the end of this week,
> so don't hold your breath on replies from me if you post
> after, let's say, thursday.
>
> -----
> PEP: 298
> Title: The Fixed Buffer Interface
> Version: $Revision: 1.3 $
> Last-Modified: $Date: 2002/07/30 16:52:53 $
> Author: Thomas Heller <theller@python.net>
> Status: Draft
> Type: Standards Track
> Created: 26-Jul-2002
> Python-Version: 2.3
> Post-History:
>
>
> Abstract
>
> This PEP proposes an extension to the buffer interface called the
> 'fixed buffer interface'.
>
> The fixed buffer interface fixes the flaws of the 'old' buffer
> interface as defined in Python versions up to and including 2.2,
> see [1]:
(I keep reading this backwards, thinking that the following two items
list the flaws in [1]. :-)
> The lifetime of the retrieved pointer is clearly defined.
>
> The buffer size is returned as a 'size_t' data type, which
> allows access to large buffers on platforms where sizeof(int)
> != sizeof(void *).
This second sounds like a change we could also make to the "old"
buffer interface, if we introduce another flag bit that's *not* part
of the default flags.
> Specification
>
> The fixed buffer interface exposes new functions which return the
> size and the pointer to the internal memory block of any python
> object which chooses to implement this interface.
>
> The size and pointer returned must be valid as long as the object
> is alive (has a positive reference count). So, only objects which
> never reallocate or resize the memory block are allowed to
> implement this interface.
>
> The fixed buffer interface omits the memory segment model which is
> present in the old buffer interface - only a single memory block
> can be exposed.
>
>
> Implementation
>
> Define a new flag in Include/object.h:
>
> /* PyBufferProcs contains bf_getfixedreadbuffer
> and bf_getfixedwritebuffer */
> #define Py_TPFLAGS_HAVE_GETFIXEDBUFFER (1L<<15)
>
>
> This flag would be included in Py_TPFLAGS_DEFAULT:
>
> #define Py_TPFLAGS_DEFAULT ( \
> ....
> Py_TPFLAGS_HAVE_GETFIXEDBUFFER | \
> ....
> 0)
>
>
> Extend the PyBufferProcs structure by new fields in
> Include/object.h:
>
> typedef size_t (*getfixedreadbufferproc)(PyObject *, void **);
> typedef size_t (*getfixedwritebufferproc)(PyObject *, void **);
>
> typedef struct {
> getreadbufferproc bf_getreadbuffer;
> getwritebufferproc bf_getwritebuffer;
> getsegcountproc bf_getsegcount;
> getcharbufferproc bf_getcharbuffer;
> /* fixed buffer interface functions */
> getfixedreadbufferproc bf_getfixedreadbufferproc;
> getfixedwritebufferproc bf_getfixedwritebufferproc;
> } PyBufferProcs;
>
>
> The new fields are present if the Py_TPFLAGS_HAVE_GETFIXEDBUFFER
> flag is set in the object's type.
>
> The Py_TPFLAGS_HAVE_GETFIXEDBUFFER flag implies the
> Py_TPFLAGS_HAVE_GETCHARBUFFER flag.
>
> The getfixedreadbufferproc and getfixedwritebufferproc functions
> return the size in bytes of the memory block on success, and fill
> in the passed void * pointer on success. If these functions fail
> - either because an error occurs or no memory block is exposed -
> they must set the void * pointer to NULL and raise an exception.
> The return value is undefined in these cases and should not be
> used.
>
> Usually the getfixedwritebufferproc and getfixedreadbufferproc
> functions aren't called directly, they are called through
> convenience functions declared in Include/abstract.h:
>
> int PyObject_AsFixedReadBuffer(PyObject *obj,
> void **buffer,
> size_t *buffer_len);
>
> int PyObject_AsFixedWriteBuffer(PyObject *obj,
> void **buffer,
> size_t *buffer_len);
>
> These functions return 0 on success, set buffer to the memory
> location and buffer_len to the length of the memory block in
> bytes. On failure, or if the fixed buffer interface is not
> implemented by obj, they return -1 and set an exception.
>
>
> Backward Compatibility
>
> The size of the PyBufferProcs structure changes if this proposal
> is implemented, but the type's tp_flags slot can be used to
> determine if the additional fields are present.
>
>
> Reference Implementation
>
> Will be uploaded to the SourceForge patch manager by the author.
I'm holding my breath now...
>
> Additional Notes/Comments
>
> Python strings, Unicode strings, mmap objects, and maybe other
> types would expose the fixed buffer interface, but the array type
> would *not*, because its memory block may be reallocated during
> its lifetime.
>
>
> Community Feedback
>
> Greg Ewing doubts the fixed buffer interface is needed at all, he
> thinks the normal buffer interface could be used if the pointer is
> (re)fetched each time it's used. This seems to be dangerous,
> because even innocent looking calls to the Python API like
> Py_DECREF() may trigger execution of arbitrary Python code.
>
> Neil Hodgson wants to expose pointers to memory blocks with
> limited lifetime: do some kind of lock operation on the object,
> retrieve the pointer, use it, and unlock the object again. While
> the author sees the need for this, it cannot be addressed by this
> proposal. Beeing required to call a function after not using the
x
> pointer received by the getfixedbufferprocs any more seems too
> error prone.
>
>
> Credits
>
> Scott Gilbert came up with the name 'fixed buffer interface'.
>
>
> References
>
> [1] The buffer interface
> http://mail.python.org/pipermail/python-dev/2000-October/009974.html
>
> [2] The Buffer Problem
> http://www.python.org/peps/pep-0296.html
>
>
> Copyright
>
> This document has been placed in the public domain.
>
>
>
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> End:
--Guido van Rossum (home page: http://www.python.org/~guido/)