[Python-Dev] ctypes, memory mapped files and context manager

[eryk sun]

On Thu, Jan 5, 2017 at 2:37 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 5 January 2017 at 10:28, Hans-Peter Jansen <hpj at urpla.net> wrote:
>> In order to get this working properly, the ctypes mapping needs a method to
>> free the mapping actively. E.g.:
>>
>> @contextmanager
>> def map_struct(m, n):
>>     m.resize(n * mmap.PAGESIZE)
>>     yield T.from_buffer(m)
>>          T.unmap_buffer(m)
>>
>> Other attempts with weakref and the like do not work due to the nature of the
>> ctypes types.
>
> I don't know ctypes well enough myself to comment on the idea of
> offering fully deterministic cleanup, but the closest you could get to
> that without requiring a change to ctypes is to have the context
> manager introduce a layer of indirection:

I think that's the best you can do with the current state of ctypes.

from_buffer was made safer in Python 3 by ensuring it keeps a
memoryview reference in the _objects attribute (i.e.
CDataObject.b_objects). Hans-Peter's problem is a consequence of this
reference. Simply calling release() on the underlying memoryview is
unsafe. For example:

    >>> b = bytearray(2**20)
    >>> a = ctypes.c_char.from_buffer(b)
    >>> a._objects
    <memory at 0x7f04283b8dc8>
    >>> a._objects.release()
    >>> del b
    >>> a.value
    Segmentation fault (core dumped)

A release() method on ctypes objects could release the memoryview and
also clear the CDataObject b_ptr field. In this case, any function
that accesses b_ptr would have to be modified to raise a ValueError
for a NULL value. Currently ctypes assumes b_ptr is valid, so this
would require adding a lot of checks.

On a related note, ctypes objects aren't tracking the number of
exported views like they should. resize() should raise a BufferError
in the following example:

    >>> b = (ctypes.c_char * (2**20))(255)
    >>> m = memoryview(b).cast('B')
    >>> m[0]
    255
    >>> ctypes.resize(b, 2**22)
    >>> m[0]
    Segmentation fault (core dumped)