Sharing Data in Python

[Jean-Michel Pichavant]

raunakgup90 at gmail.com wrote:
> I have some Pickled data, which is stored on disk, and it is about 100 MB in size.
>
> When my python program is executed, the picked data is loaded using the cPickle module, and all that works fine.
>
> If I execute the python multiple times using python main.py for example, each python process will load the same data multiple times, which is the correct behaviour.
>
> How can I make it so, all new python process share this data, so it is only loaded a single time into memory?
>
> asked the same question on SO, but could not get any constructive responses.. http://stackoverflow.com/questions/10550870/sharing-data-in-python/10551845
>   
Well a straightforward way of solving this is to use threads.

Have your main program spawn threads for processing data.
Now what triggers a thread is up to you, your main program can listen to 
commands, or catch keyboard events ...

If you want to make sure your using all CPU cores, you may want to use 
the module
http://docs.python.org/library/multiprocessing.html

It emulates threads with subprocesses, and features a way to share memory.
I do tend to prefer processes over threads, I find them easier to 
monitor and control.

Cheers,

JM