ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

Dear Peter,

thanks for taking valuable time to look into my issue.

It might be related to my distinct silliness, but the problem persists with 
your code as well. Further comments inlined.  

On Dienstag, 27. Dezember 2016 21:39:51 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > 
> >     def __enter__(self):
> >         # resize the mmap (and backing file), if structure exceeds mmap
> >         # size mmap size must be aligned to mmap.PAGESIZE
> >         cssize = ctypes.sizeof(self._cstruct)
> >         
> >         if self._offset + cssize > self._mm.size():
> >             newsize = align(self._offset + cssize, mmap.PAGESIZE)
> >             self._mm.resize(newsize)
> >         
> >         self._csinst = self._cstruct.from_buffer(self._mm, self._offset)
> >         return self._csinst
> 
> Here you give away a reference to the ctypes.BigEndianStructure. That means
> you no longer control the lifetime of self._csinst which in turn holds a
> reference to the underlying mmap or whatever it's called.

Here's the code, based on contextmanager:

@contextmanager
def cstructmap(cstruct, mm, offset = 0):
    # resize the mmap (and backing file), if structure exceeds mmap size
    # mmap size must be aligned to mmap.PAGESIZE
    cssize = ctypes.sizeof(cstruct)
    if offset + cssize > mm.size():
        newsize = align(offset + cssize, mmap.PAGESIZE)
        mm.resize(newsize)
    yield cstruct.from_buffer(mm, offset)

While much more concise, I don't understand, how it should make a difference 
relative to the "with" variable lifetime, when used. 
 
> There might be a way to release the mmap reference while the wrapper
> structure is still alive, but the cleaner way is probably to not give it
> away in the first place, and create a proxy instead with
> 
>           return weakref.proxy(self._csinst)

This fails, as it doesn't keep the reference long enough.

> >     def __exit__(self, exc_type, exc_value, exc_traceback):
> >         # free all references into mmap
> >         del self._csinst
> 
> The line above is redundant. It removes the attribute from the instance
> __dict__ and implicitly decreases its refcount. It does not actually
> physically delete the referenced object. If you remove the del statement the
> line below will still decrease the refcount.
> 
> Make sure you understand this to avoid littering your code with cargo cult
> del-s ;)

Yes, I was aware of this. It was a testing relic, that survived somehow. 
Sorry. Yes, I usually try to avoid cargo cultry in my code. ;)

> > The issue: when creating a mapping via context manager, we assign a local
> > variable (with ..), that keep existing in the local context, even when the
> > manager context was left. This keeps a reference on the ctypes mapped area
> > alive, even if we try everything to destroy it in __exit__. We have to del
> > the with var manually.
> > 
> > Now, I want to get rid of the ugly any error prone del statements.
> > 
> > What is needed, is a ctypes operation, that removes the mapping actively,
> > and that could be added to the __exit__ part of the context manager.

Revised code (including your test code):
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239

> > The script creates a memory mapped file in the current directory named
> > "mapfile". When started without arguments, it copies itself into this
> > file, until 10 * mmap.PAGESIZE growth is reached (or it errored out
> > before..).
> > 
> > IF you change NOPROB to True, it will actively destruct the context
> > manager vars, and should work as advertized.
> > 
> > Any ideas are much appreciated.
> 
> You might put some more effort into composing example scripts. Something
> like the script below would have saved me some time...

I'm very sorry about this. 
 
> import ctypes
> import mmap
> 
> from contextlib import contextmanager
> 
> class T(ctypes.Structure):
>     _fields = [("foo", ctypes.c_uint32)]
> 
> 
> @contextmanager
> def map_struct(m, n):
>     m.resize(n * mmap.PAGESIZE)
>     yield T.from_buffer(m)
> 
> SIZE = mmap.PAGESIZE * 2
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
> m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
> 
> with map_struct(m, 1) as a:
>     a.foo = 1
> with map_struct(m, 2) as b:
>     b.foo = 2

Unfortunately, your code behaves exactly like mine:

$> python3 mmap_test.py
Traceback (most recent call last):
  File "mmap_test.py", line 23, in <module>
    with map_struct(m, 2) as b:
  File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "mmap_test.py", line 12, in map_struct
    m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.

BTW, Python2 makes a difference in this respect, but my project is Python3 
based. Have you tested this with Python3? It would be interesting to explore 
the reasons of this difference, which is, ähem, pretty surprising.

Thanks,
Pete