"multiprocessing.shared_memory" — Shared memory for direct access across processes
**********************************************************************************

**Source code:** Lib/multiprocessing/shared_memory.py

Added in version 3.8.

======================================================================

This module provides a class, "SharedMemory", for the allocation and
management of shared memory to be accessed by one or more processes on
a multicore or symmetric multiprocessor (SMP) machine.  To assist with
the life-cycle management of shared memory especially across distinct
processes, a "BaseManager" subclass, "SharedMemoryManager", is also
provided in the "multiprocessing.managers" module.

In this module, shared memory refers to “POSIX style” shared memory
blocks (though is not necessarily implemented explicitly as such) and
does not refer to “distributed shared memory”.  This style of shared
memory permits distinct processes to potentially read and write to a
common (or shared) region of volatile memory.  Processes are
conventionally limited to only have access to their own process memory
space but shared memory permits the sharing of data between processes,
avoiding the need to instead send messages between processes
containing that data.  Sharing data directly via memory can provide
significant performance benefits compared to sharing data via disk or
socket or other communications requiring the
serialization/deserialization and copying of data.

class multiprocessing.shared_memory.SharedMemory(name=None, create=False, size=0, *, track=True)

   Create an instance of the "SharedMemory" class for either creating
   a new shared memory block or attaching to an existing shared memory
   block.  Each shared memory block is assigned a unique name. In this
   way, one process can create a shared memory block with a particular
   name and a different process can attach to that same shared memory
   block using that same name.

   As a resource for sharing data across processes, shared memory
   blocks may outlive the original process that created them.  When
   one process no longer needs access to a shared memory block that
   might still be needed by other processes, the "close()" method
   should be called. When a shared memory block is no longer needed by
   any process, the "unlink()" method should be called to ensure
   proper cleanup.

   Parameters:
      * **name** (*str** | **None*) – The unique name for the
        requested shared memory, specified as a string. When creating
        a new shared memory block, if "None" (the default) is supplied
        for the name, a novel name will be generated.

      * **create** (*bool*) – Control whether a new shared memory
        block is created ("True") or an existing shared memory block
        is attached ("False").

      * **size** (*int*) – The requested number of bytes when creating
        a new shared memory block. Because some platforms choose to
        allocate chunks of memory based upon that platform’s memory
        page size, the exact size of the shared memory block may be
        larger or equal to the size requested. When attaching to an
        existing shared memory block, the *size* parameter is ignored.

      * **track** (*bool*) – When "True", register the shared memory
        block with a resource tracker process on platforms where the
        OS does not do this automatically. The resource tracker
        ensures proper cleanup of the shared memory even if all other
        processes with access to the memory exit without doing so.
        Python processes created from a common ancestor using
        "multiprocessing" facilities share a single resource tracker
        process, and the lifetime of shared memory segments is handled
        automatically among these processes. Python processes created
        in any other way will receive their own resource tracker when
        accessing shared memory with *track* enabled. This will cause
        the shared memory to be deleted by the resource tracker of the
        first process that terminates. To avoid this issue, users of
        "subprocess" or standalone Python processes should set *track*
        to "False" when there is already another process in place that
        does the bookkeeping. *track* is ignored on Windows, which has
        its own tracking and automatically deletes shared memory when
        all handles to it have been closed.

   Changed in version 3.13: Added the *track* parameter.

   close()

      Close the file descriptor/handle to the shared memory from this
      instance.  "close()" should be called once access to the shared
      memory block from this instance is no longer needed.  Depending
      on operating system, the underlying memory may or may not be
      freed even if all handles to it have been closed.  To ensure
      proper cleanup, use the "unlink()" method.

   unlink()

      Delete the underlying shared memory block.  This should be
      called only once per shared memory block regardless of the
      number of handles to it, even in other processes. "unlink()" and
      "close()" can be called in any order, but trying to access data
      inside a shared memory block after "unlink()" may result in
      memory access errors, depending on platform.

      This method has no effect on Windows, where the only way to
      delete a shared memory block is to close all handles.

   buf

      A memoryview of contents of the shared memory block.

   name

      Read-only access to the unique name of the shared memory block.

   size

      Read-only access to size in bytes of the shared memory block.

The following example demonstrates low-level use of "SharedMemory"
instances:

   >>> from multiprocessing import shared_memory
   >>> shm_a = shared_memory.SharedMemory(create=True, size=10)
   >>> type(shm_a.buf)
   <class 'memoryview'>
   >>> buffer = shm_a.buf
   >>> len(buffer)
   10
   >>> buffer[:4] = bytearray([22, 33, 44, 55])  # Modify multiple at once
   >>> buffer[4] = 100                           # Modify single byte at a time
   >>> # Attach to an existing shared memory block
   >>> shm_b = shared_memory.SharedMemory(shm_a.name)
   >>> import array
   >>> array.array('b', shm_b.buf[:5])  # Copy the data into a new array.array
   array('b', [22, 33, 44, 55, 100])
   >>> shm_b.buf[:5] = b'howdy'  # Modify via shm_b using bytes
   >>> bytes(shm_a.buf[:5])      # Access via shm_a
   b'howdy'
   >>> shm_b.close()   # Close each SharedMemory instance
   >>> shm_a.close()
   >>> shm_a.unlink()  # Call unlink only once to release the shared memory

The following example demonstrates a practical use of the
"SharedMemory" class with NumPy arrays, accessing the same
"numpy.ndarray" from two distinct Python shells:

   >>> # In the first Python interactive shell
   >>> import numpy as np
   >>> a = np.array([1, 1, 2, 3, 5, 8])  # Start with an existing NumPy array
   >>> from multiprocessing import shared_memory
   >>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
   >>> # Now create a NumPy array backed by shared memory
   >>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
   >>> b[:] = a[:]  # Copy the original data into shared memory
   >>> b
   array([1, 1, 2, 3, 5, 8])
   >>> type(b)
   <class 'numpy.ndarray'>
   >>> type(a)
   <class 'numpy.ndarray'>
   >>> shm.name  # We did not specify a name so one was chosen for us
   'psm_21467_46075'

   >>> # In either the same shell or a new Python shell on the same machine
   >>> import numpy as np
   >>> from multiprocessing import shared_memory
   >>> # Attach to the existing shared memory block
   >>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075')
   >>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example
   >>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf)
   >>> c
   array([1, 1, 2, 3, 5, 8])
   >>> c[-1] = 888
   >>> c
   array([  1,   1,   2,   3,   5, 888])

   >>> # Back in the first Python interactive shell, b reflects this change
   >>> b
   array([  1,   1,   2,   3,   5, 888])

   >>> # Clean up from within the second Python shell
   >>> del c  # Unnecessary; merely emphasizing the array is no longer used
   >>> existing_shm.close()

   >>> # Clean up from within the first Python shell
   >>> del b  # Unnecessary; merely emphasizing the array is no longer used
   >>> shm.close()
   >>> shm.unlink()  # Free and release the shared memory block at the very end

class multiprocessing.managers.SharedMemoryManager([address[, authkey]])

   A subclass of "multiprocessing.managers.BaseManager" which can be
   used for the management of shared memory blocks across processes.

   A call to "start()" on a "SharedMemoryManager" instance causes a
   new process to be started. This new process’s sole purpose is to
   manage the life cycle of all shared memory blocks created through
   it.  To trigger the release of all shared memory blocks managed by
   that process, call "shutdown()" on the instance. This triggers a
   "unlink()" call on all of the "SharedMemory" objects managed by
   that process and then stops the process itself.  By creating
   "SharedMemory" instances through a "SharedMemoryManager", we avoid
   the need to manually track and trigger the freeing of shared memory
   resources.

   This class provides methods for creating and returning
   "SharedMemory" instances and for creating a list-like object
   ("ShareableList") backed by shared memory.

   Refer to "BaseManager" for a description of the inherited *address*
   and *authkey* optional input arguments and how they may be used to
   connect to an existing "SharedMemoryManager" service from other
   processes.

   SharedMemory(size)

      Create and return a new "SharedMemory" object with the specified
      *size* in bytes.

   ShareableList(sequence)

      Create and return a new "ShareableList" object, initialized by
      the values from the input *sequence*.

The following example demonstrates the basic mechanisms of a
"SharedMemoryManager":

   >>> from multiprocessing.managers import SharedMemoryManager
   >>> smm = SharedMemoryManager()
   >>> smm.start()  # Start the process that manages the shared memory blocks
   >>> sl = smm.ShareableList(range(4))
   >>> sl
   ShareableList([0, 1, 2, 3], name='psm_6572_7512')
   >>> raw_shm = smm.SharedMemory(size=128)
   >>> another_sl = smm.ShareableList('alpha')
   >>> another_sl
   ShareableList(['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221')
   >>> smm.shutdown()  # Calls unlink() on sl, raw_shm, and another_sl

The following example depicts a potentially more convenient pattern
for using "SharedMemoryManager" objects via the "with" statement to
ensure that all shared memory blocks are released after they are no
longer needed:

   >>> with SharedMemoryManager() as smm:
   ...     sl = smm.ShareableList(range(2000))
   ...     # Divide the work among two processes, storing partial results in sl
   ...     p1 = Process(target=do_work, args=(sl, 0, 1000))
   ...     p2 = Process(target=do_work, args=(sl, 1000, 2000))
   ...     p1.start()
   ...     p2.start()  # A multiprocessing.Pool might be more efficient
   ...     p1.join()
   ...     p2.join()   # Wait for all work to complete in both processes
   ...     total_result = sum(sl)  # Consolidate the partial results now in sl

When using a "SharedMemoryManager" in a "with" statement, the shared
memory blocks created using that manager are all released when the
"with" statement’s code block finishes execution.

class multiprocessing.shared_memory.ShareableList(sequence=None, *, name=None)

   Provide a mutable list-like object where all values stored within
   are stored in a shared memory block. This constrains storable
   values to the following built-in data types:

   * "int" (signed 64-bit)

   * "float"

   * "bool"

   * "str" (less than 10M bytes each when encoded as UTF-8)

   * "bytes" (less than 10M bytes each)

   * "None"

   It also notably differs from the built-in "list" type in that these
   lists can not change their overall length (i.e. no "append()",
   "insert()", etc.) and do not support the dynamic creation of new
   "ShareableList" instances via slicing.

   *sequence* is used in populating a new "ShareableList" full of
   values. Set to "None" to instead attach to an already existing
   "ShareableList" by its unique shared memory name.

   *name* is the unique name for the requested shared memory, as
   described in the definition for "SharedMemory".  When attaching to
   an existing "ShareableList", specify its shared memory block’s
   unique name while leaving *sequence* set to "None".

   Note:

     A known issue exists for "bytes" and "str" values. If they end
     with "\x00" nul bytes or characters, those may be *silently
     stripped* when fetching them by index from the "ShareableList".
     This ".rstrip(b'\x00')" behavior is considered a bug and may go
     away in the future. See gh-106939.

   For applications where rstripping of trailing nulls is a problem,
   work around it by always unconditionally appending an extra non-0
   byte to the end of such values when storing and unconditionally
   removing it when fetching:

      >>> from multiprocessing import shared_memory
      >>> nul_bug_demo = shared_memory.ShareableList(['?\x00', b'\x03\x02\x01\x00\x00\x00'])
      >>> nul_bug_demo[0]
      '?'
      >>> nul_bug_demo[1]
      b'\x03\x02\x01'
      >>> nul_bug_demo.shm.unlink()
      >>> padded = shared_memory.ShareableList(['?\x00\x07', b'\x03\x02\x01\x00\x00\x00\x07'])
      >>> padded[0][:-1]
      '?\x00'
      >>> padded[1][:-1]
      b'\x03\x02\x01\x00\x00\x00'
      >>> padded.shm.unlink()

   count(value)

      Return the number of occurrences of *value*.

   index(value)

      Return first index position of *value*. Raise "ValueError" if
      *value* is not present.

   format

      Read-only attribute containing the "struct" packing format used
      by all currently stored values.

   shm

      The "SharedMemory" instance where the values are stored.

The following example demonstrates basic use of a "ShareableList"
instance:

>>> from multiprocessing import shared_memory
>>> a = shared_memory.ShareableList(['howdy', b'HoWdY', -273.154, 100, None, True, 42])
>>> [ type(entry) for entry in a ]
[<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>]
>>> a[2]
-273.154
>>> a[2] = -78.5
>>> a[2]
-78.5
>>> a[2] = 'dry ice'  # Changing data types is supported as well
>>> a[2]
'dry ice'
>>> a[2] = 'larger than previously allocated storage space'
Traceback (most recent call last):
  ...
ValueError: exceeds available storage for existing str
>>> a[2]
'dry ice'
>>> len(a)
7
>>> a.index(42)
6
>>> a.count(b'howdy')
0
>>> a.count(b'HoWdY')
1
>>> a.shm.close()
>>> a.shm.unlink()
>>> del a  # Use of a ShareableList after call to unlink() is unsupported

The following example depicts how one, two, or many processes may
access the same "ShareableList" by supplying the name of the shared
memory block behind it:

>>> b = shared_memory.ShareableList(range(5))         # In a first process
>>> c = shared_memory.ShareableList(name=b.shm.name)  # In a second process
>>> c
ShareableList([0, 1, 2, 3, 4], name='...')
>>> c[-1] = -999
>>> b[-1]
-999
>>> b.shm.close()
>>> c.shm.close()
>>> c.shm.unlink()

The following examples demonstrates that "ShareableList" (and
underlying "SharedMemory") objects can be pickled and unpickled if
needed. Note, that it will still be the same shared object. This
happens, because the deserialized object has the same unique name and
is just attached to an existing object with the same name (if the
object is still alive):

>>> import pickle
>>> from multiprocessing import shared_memory
>>> sl = shared_memory.ShareableList(range(10))
>>> list(sl)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> deserialized_sl = pickle.loads(pickle.dumps(sl))
>>> list(deserialized_sl)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> sl[0] = -1
>>> deserialized_sl[1] = -2
>>> list(sl)
[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(deserialized_sl)
[-1, -2, 2, 3, 4, 5, 6, 7, 8, 9]

>>> sl.shm.close()
>>> sl.shm.unlink()
