Multiprocessing, shared memory vs. pickled copies

[Robert Kern]

On 4/4/11 7:05 PM, Robert Kern wrote:
> On 4/4/11 3:20 PM, John Ladasky wrote:

> However, at least in Python 2.7, multiprocessing seems to have a C extension
> module defining the Connection objects. Unfortunately, it looks like this C
> extension just imports the regular pickler that is not aware of these custom
> types. That's why you get this error. I believe this is a bug in Python.
>
> So what did you try, and what output did you get? What version of Python are you
> using?
>
>> I think that my understanding of multiprocessing
>> needs to evolve beyond the use of Pool, but I'm not sure yet. This
>> post suggests as much.
>>
>> http://mail.scipy.org/pipermail/scipy-user/2009-February/019696.html
>
> Maybe. If the __reduce_ex__() method is implemented properly (and
> multiprocessing bugs aren't getting in the way), you ought to be able to pass
> them to a Pool just fine. You just need to make sure that the shared arrays are
> allocated before the Pool is started. And this only works on UNIX machines. The
> shared memory objects that shmarray uses can only be inherited. I believe that's
> what Sturla was getting at.

I'll take that back a little bit. Since the underlying shared memory types can 
only be shared by inheritance, the ForkingPickler is only used for the arguments 
passed to Process(), and not used for things sent through Queues (which Pool 
uses). Since Pool cannot guarantee that the data exists before the Pool starts 
its subprocesses, it must use the general mechanism.

So in short, if you pass the shmarrays as arguments to the target function in 
Process(), it should work fine.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco