Multiprocessing, shared memory vs. pickled copies

[sturlamolden]

On 11 apr, 09:21, John Nagle <na... at animats.com> wrote:

>      Because nobody can fix the Global Interpreter Lock problem in CPython.
>
>      The multiprocessing module is a hack to get around the fact
> that Python threads don't run concurrently, and thus, threaded
> programs don't effectively use multi-core CPUs.

Then why not let NumPy use shared memory in this case? It is as simple
as this:

import numpy as np
import sharedmem as sm
private_array = np.zeros((10,10))
shared_array  = sm.zeros((10,10))

I.e. the code is identical (more or less).

Also, do you think shared memory has other uses besides parallel
computing, such as IPC? It is e.g. an easy way to share data between
Python and an external program. Do you e.g. need to share (not send)
large amounts of data between Python and Java or Matlab?

The difference here from multiprocessing.Array is that IPC with
multiprocessing.Queue will actually work, because I am not using
anonymous shared memory. Instead of inherting handles, the segment has
a name in the file system, which means it can be memory mapped from
anywhere.

I posted the code to the NumPy mailing list yesterday.

Sturla