ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> > > Hans-Peter Jansen wrote:
> > > > Dear Peter,
> > > > 
> > > > thanks for taking valuable time to look into my issue.
> > > 
> > > You're welcome!
> > > 
> > > @contextmanager
> > > def map_struct(m, n):
> > >     m.resize(n * mmap.PAGESIZE)
> > >     keep_me = T.from_buffer(m)
> > >     yield weakref.proxy(keep_me)
> > 
> > Hooray, that did the trick. Great solution, thank you very much!
> 
> Sorry for bothering you again, Peter, but after applying it to the real
> project, that fails consistently similar to:

$ python3 mmap_test_weakref.py 
Traceback (most recent call last):
  File "mmap_test_weakref.py", line 32, in <module>
    assert(bytes(c) == bytes(rest))
AttributeError: 'c_ubyte_Array_8188' object has no attribute '__bytes__'


$ cat mmap_test_weakref.py
import ctypes
import mmap
import weakref

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields_ = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n, struct, offset = 0):
    m.resize(n * mmap.PAGESIZE)
    inst = struct.from_buffer(m, offset)
    yield weakref.proxy(inst)

SIZE = mmap.PAGESIZE
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1, T) as a:
    a.foo = 1
with map_struct(m, 2, T) as b:
    b.foo = 2

offset = ctypes.sizeof(T)
rest = m.size() - offset
overhang = ctypes.c_ubyte * rest
with map_struct(m, 2, overhang, offset) as c:
    assert(bytes(c) == bytes(rest))


With weakref and mmap.resize() disabled, this acts as expected.
BTW: mmapped files need the first page initialized, the rest is done in the 
kernel (filled with zeros on resize).

Pete