ctypes, memory mapped files and context manager

[Peter Otten]

Hans-Peter Jansen wrote:

> On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote:
>> On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
>> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
>> > > Hans-Peter Jansen wrote:
>> > > > Dear Peter,
>> > > > 
>> > > > thanks for taking valuable time to look into my issue.
>> > > 
>> > > You're welcome!
>> > > 
>> > > @contextmanager
>> > > def map_struct(m, n):
>> > >     m.resize(n * mmap.PAGESIZE)
>> > >     keep_me = T.from_buffer(m)
>> > >     yield weakref.proxy(keep_me)
>> > 
>> > Hooray, that did the trick. Great solution, thank you very much!
>> 
>> Sorry for bothering you again, Peter, but after applying it to the real
>> project, that fails consistently similar to:
> 
> $ python3 mmap_test_weakref.py
> Traceback (most recent call last):
>   File "mmap_test_weakref.py", line 32, in <module>
>     assert(bytes(c) == bytes(rest))
> AttributeError: 'c_ubyte_Array_8188' object has no attribute '__bytes__'
> 
> 
> $ cat mmap_test_weakref.py
> import ctypes
> import mmap
> import weakref
> 
> from contextlib import contextmanager
> 
> class T(ctypes.Structure):
>     _fields_ = [("foo", ctypes.c_uint32)]
> 
> 
> @contextmanager
> def map_struct(m, n, struct, offset = 0):
>     m.resize(n * mmap.PAGESIZE)
>     inst = struct.from_buffer(m, offset)
>     yield weakref.proxy(inst)
> 
> SIZE = mmap.PAGESIZE
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
> m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
> 
> with map_struct(m, 1, T) as a:
>     a.foo = 1
> with map_struct(m, 2, T) as b:
>     b.foo = 2
> 
> offset = ctypes.sizeof(T)
> rest = m.size() - offset
> overhang = ctypes.c_ubyte * rest
> with map_struct(m, 2, overhang, offset) as c:
>     assert(bytes(c) == bytes(rest))
> 
> 
> With weakref and mmap.resize() disabled, this acts as expected.
> BTW: mmapped files need the first page initialized, the rest is done in
> the kernel (filled with zeros on resize).

The minimal example is

>>> import weakref, ctypes
>>> T = ctypes.c_ubyte * 3
>>> t = T()
>>> bytes(t) == b"\0" * 3
True
>>> bytes(weakref.proxy(t)) == b"\0" * 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'

That looks like a leaky abstraction. While I found a workaround

>>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
True

to me your whole approach is beginning to look very questionable. You know, 

"If the implementation is hard to explain, it's a bad idea."

What do you gain from using the mmap/ctypes combo instead of regular file 
operations and the struct module? Your sample code seems to touch every 
single byte of the file once so that there are little to no gains from 
caching. And then your offset is basically a file position managed manually 
instead of implicitly with read, write, and seek.