[Python-Dev] ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

Hi Eryk,

On Donnerstag, 5. Januar 2017 15:30:33 eryk sun wrote:
> 
> > manager introduce a layer of indirection:
> I think that's the best you can do with the current state of ctypes.
> 
> from_buffer was made safer in Python 3 by ensuring it keeps a
> memoryview reference in the _objects attribute (i.e.
> CDataObject.b_objects). Hans-Peter's problem is a consequence of this
> reference. Simply calling release() on the underlying memoryview is
> 
> unsafe. For example:
>     >>> b = bytearray(2**20)
>     >>> a = ctypes.c_char.from_buffer(b)
>     >>> a._objects
> 
>     <memory at 0x7f04283b8dc8>
> 
>     >>> a._objects.release()
>     >>> del b
>     >>> a.value
> 
>     Segmentation fault (core dumped)

This is exactly, what I was after:

@contextmanager
def cstructmap(cstruct, mm, offset = 0):
    # resize the mmap (and backing file), if structure exceeds mmap size
    # mmap size must be aligned to mmap.PAGESIZE
    cssize = ctypes.sizeof(cstruct)
    if offset + cssize > mm.size():
        newsize = align(offset + cssize, mmap.PAGESIZE)
        mm.resize(newsize)
    cmap = cstruct.from_buffer(mm, offset)
    try:
        yield cmap
    finally:
        for mv in cmap._objects.values():
            mv.release()

See also:
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239#file-ctypes_mmap_ctx3-py

While technically possible (which is a surprise for me on its own), nothing 
should access the with variable after the block has finished. If that happens, 
a segfault is exactly was it deserves IMHO.

Leaves the question, how stable this "interface" is?
Accessing _objects here belongs to voodoo programming practices of course, but 
the magic is locally limited to just two lines of code, which is acceptable in 
order to get this context manager working without messing with the rest of the 
code.

Opinions?

Thanks,
Pete