[issue35813] shared memory construct to avoid need for serialization between processes

[Davin Potts]

Davin Potts <python at discontinuity.net> added the comment:

@giampaolo:

> Also, what happens if you alter the size of an existing object with a smaller value? Is the memory region overwritten?

Attaching to an existing shared memory block with a size=N which is smaller than its allocated size (say it was created with size=M and N<M) will succeed in giving you access to at least the first N bytes of that shared memory block.  The shared memory block allocated size will still be M bytes.  That means future attempts to attach to it with size=M will continue to work as well.  One motivation for why this is supported is, I believe, to deliberately limit access to part of the shared memory in some parts of a user's code, avoiding potential coding mistakes by design; for example, say the first 100KB contains unchanging reference data that many processes need to access but the second 100KB contains rapidly changing data that only a few processes should ever access or manipulate.  I expect the most common use is to simply attach to the whole of a shared memory block, but I would not want to unnecessarily limit other established use cases.

This behavior needed to be captured in the docs but I see it has not been!  I have now added to the description of the size parameter and it should show up in GH-11816 shortly.


> Can't you just avoid calling ftruncate() if size is not passed (None)?

It looks like it does skip calling ftruncate() if size is 0.  From posixshmem.c:
        if (size) {
            DPRINTF("calling ftruncate, fd = %d, size = %ld\n", self->fd, size);
            if (-1 == ftruncate(self->fd, (off_t)size)) {


>> I think this misses the ...
> It appears this is already covered:

Sorry for any confusion; I was interpreting your proposed parameter name, attach_if_exists, in the following way:
* attach_if_exists=True:   If exists, attach to it otherwise create one
* attach_if_exists=False:  Create a new one but do not attach to an existing with the same name
I did not see a way to indicate a desire to *only* attach without creation.  I need a way to test to see if a shared memory block already exists or not without risk of creating one.  At least this is how I was interpreting "attach if exists".


> Don't you also want to "create if it doesn't exist, else attach" as a single, atomic operation?

Yes, I do!  This was part of my description for the parameter named "create" in msg335660:
    When set to True, a new shared memory block will be created unless
    one already exists with the supplied unique name, in which case that block
    will be attached to and used.


> I'm not sure if there are or should be sync primitives to "wait for another memory to join me" etc.

In the case of shared memory, I do not think so.  I think such signaling between processes, when needed, can be accomplished by our existing signaling mechanisms (like, via the Proxy Objects for Event or Semaphore).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35813>
_______________________________________