ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

Dear Eryk,

thanks for chiming in.

On Donnerstag, 29. Dezember 2016 21:27:56 eryk sun wrote:
> On Thu, Dec 29, 2016 at 12:18 PM, Hans-Peter Jansen <hpj at urpla.net> wrote:
> >> >>> import weakref, ctypes
> >> >>> T = ctypes.c_ubyte * 3
> >> >>> t = T()
> >> >>> bytes(t) == b"\0" * 3
> >> 
> >> True
> >> 
> >> >>> bytes(weakref.proxy(t)) == b"\0" * 3
> >> 
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >> 
> >> AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'
> >> 
> >> That looks like a leaky abstraction. While I found a workaround
> >> 
> >> >>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
> >> 
> >> True
> > 
> > I found a couple of other rough corners already, when working with the
> > ctypes module. Obviously, this module is lacking some love.
> 
> That's not the fault of ctypes. There's no requirement for objects
> that implement the buffer protocol to also implement __bytes__. You'd
> have the same problem if you tried to proxy a memoryview.
> 
> However, using a proxy seems particularly wHaorthless for ctypes. Type
> checking is integral to the design of ctypes, and a weakproxy won't

Did you follow the discussion? 

I'm trying to make context manager work with ctypes.from_buffer on mmapped 
files:

import ctypes
import mmap
import weakref

NOPROB=False
#NOPROB=True

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields_ = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n, struct, offset = 0):
    m.resize(n * mmap.PAGESIZE)
    inst = struct.from_buffer(m, offset)
    yield inst

SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1, T) as a:
    a.foo = 1
if NOPROB:
    del a
with map_struct(m, 2, T) as b:
    b.foo = 2
if NOPROB:
    del b

offset = ctypes.sizeof(T)
rest = m.size() - offset
overhang = ctypes.c_ubyte * rest
with map_struct(m, 2, overhang, offset) as c:
    assert(bytes(c) == bytes(rest))
if NOPROB:
    del c
 

Without these dreaded del statements, this code doesn't work:

$ python3 mmap_test2.py 
Traceback (most recent call last):
  File "mmap_test2.py", line 30, in <module>
    with map_struct(m, 2, T) as b:
  File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "mmap_test2.py", line 16, in map_struct
    m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.

It will work, if you define NOPROB=True.

The weakref approach was an attempt to make this work.

Do you have an idea how to create the context manager in a way, that obsoletes 
these ugly dels? Something like ctypes.from_buffer_release() that is able to 
actively release the mapping is needed here, AFAICS.

This code works with Python2 due to the mmap module not checking for any 
existing mappings which may lead to segfaults, if the mmap is resized.

Thanks,
Pete