[Python-ideas] multiprocessing IPC

[Sturla Molden]

Den 12.02.2012 09:02, skrev Mike Meyer:
> True, but we're talking about an API, not a specific implementation. 

You have been complaining about the GIL which is a specific implementation.

I am talking about how multiprocessing actually works, i.e. implementation.


>
>> But apart from that, show me how you would use the mmap module to make
>> named shared memory on Linux or Windows. No, memory mapping file object
>> -1 or 0 don't count, you get an anonymous memory mapping.
> The linux mmap has the same arguments as the BSD one, so I'd expect it
> to work the same.

It calls BSD mmap in the implementation on Linux. It calls 
CreateFileMapping and MapViewOfFile on Windows.


> Works exactly like I'd expect it to.
>
>> Show me how you would code this.
> Here's the code that creates the shared file:
>
>      share_name = '/tmp/xyzzy'
>      with open(share_name, 'wb') as f:
>          f.write(b'hello')
>
> Here's the code for the child:
>
>      with open(share_name, 'r+b') as f:
>          share = mmap(f.fileno(), 0)
>          share[:5] = b'gone\n'
>
> Here's the code for the parent:
>
>      child = Process(target=proc)
>      child.start()
>      with open(share_name, mode='r+b') as f:
>          share = mmap(f.fileno(), 0)
>          while share[0] == ord('h'):
>              sleep(1)
>          print('main:', share.readline())


Here you are memory mapping a temporary file, not shared memory.

On Linux, shared memory with mmap does not have a share_name.
It has fileno -1. So go ahead and replace f.fileno() with -1 and see if
it still works for you.

This is how mmap is used for shared memory on Linux:

shm = mmap.mmap(-1, 4096)
os.fork()

See how the fork comes after the mmap. Which means it must always
be allocated in the parent process. That is why we need an implementation
with System V IPC instead of mmap.



> Yes, but we're not talking about multiprocessing.Queue. We're talking 
> about mmap. multiprocessing.Queue doesn't use mmap. For that, you want 
> to us multiprocessing.Value and multiprocessing.Array.

Pass multiprocessing.Value or multiprocessing.Array to
multiprocessing.Queue and see what happens.

And while you are at it, pass multiprocessing.Lock to
multiprocessing.Queue and see what happens as well.
Contemplate how we can pass an object with a lock
as a message between two processes. Should we change
the implementation?

And then, look up the implementation for multiprocessing.Value
and Array and see if (and how) they use mmap. Perhaps you just
told me to use mmap instead of mmap.


Sturla