[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

[Davin Potts]

Davin Potts <python at discontinuity.net> added the comment:

Short responses to questions/comments from @bjs, followed by hopefully helpful further comments:

> Are you supposed to ever use a raw SharedMemory buffer directly?

Yes.


> What atomicity guarantees are there for ShareableList operations and read/write to the SharedMemory buffer?

None.


> I've had a fix spinning for about a day now, it introduced a `multiprocessing.Lock` and it was simply wrapped around any struct packing and unpacking calls.

That sounds like a nice general-purpose fix for situations where it is impossible to plan ahead to know when one or more processes will need to modify the ShareableList/SharedMemory.buf.  When it is possible to design code to ensure, because of the execution flow through the code, no two processes/threads will attempt to modify and access/modify the same location in memory at the same time, locks become unnecessary.  Locks are great tools but they generally result in slower executing code.


> What are the use cases for SharedMemory and ShareableList?

Speed.  If we don't care about speed, we can use distributed shared memory through the SyncManager -- this keeps one copy of a dict/list/whatever in the process memory space of a single process and all other processes may modify or access it through two-sided communication with that "owner" process.  If we do care about speed, we use SharedMemory and ShareableList and other things created on top of SharedMemory -- this effectively gives us fast, communal memory access where we avoid the cost of communication except for when we truly need to synchronize (where multiprocessing.Lock can help).

Reduced memory footprint.  If I have a "very large" amount of data consuming a significant percentage of the available memory on my system, I can make it available to multiple processes without duplication.  This provides processes with fast access to that data, as fast as if each were accessing data in its own process memory space.  It is one thing to imagine using this in parallel-executing code, but this can be just as useful in the Python interactive shell.  One such scenario:  after starting a time-consuming, non-parallel calculation in one Python shell, it is possible to open a new Python shell in another window and attach to the data through shared memory to continue work while the calculation runs in the first window.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37652>
_______________________________________