[Python-Dev] Do any of the 4 multiprocessing methods work using shared memory?

Kevin Ar18 kevinar18 at hotmail.com
Tue Jul 27 02:20:35 CEST 2010


Brief Summary:
Can I share Python objects between multiple processes using shared memory (using the multiprocessing module)?
In particular, do Queues or Pipes work using shared memory?

Details:
* I have several processes each on a separate CPU core (so they run in parallel).
* I want to share certain Python objects between the processes using shared memory (I can handle the locking myself -- only one process can access a variable or object at a time, so there is no conflict anyways).
* I strongly prefer that it be shared memory because I do not want to incur the costs of having to copy data back and forth between the processes.

I am aware of the multiprocessing module.  It offers:
Queues
Pipes
Shared Memory Map
Server Process

So, let me ask a few questions about those 4 items:

Queues & Pipes
These handle Python objects and variables....  However, my question is do they use shared memory or do they require copying the data or some other method that is more costly than shared memory?

Shared Memory Map
Does not support Python objects or variables -- thus no good to me.

Server Process
Looks like it requires another process that incurs extra processing just to handle the data exchange -- which, if true, is bad for me.
Does this use shared memory?  or does it require copying data back and forth between processes?
 		 	   		  
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4


More information about the Python-Dev mailing list