[SciPy-user] General parallel processing question

[Robert Kern]

Anand Patil wrote:
> Hi again everyone,
> 
> Say I have two objects A and B. A's member functions _occasionally_ want 
> to call the member functions of B, and vice versa. Both A and B have to 
> do a lot of work between calls to each other's member functions.
> 
> I'd like to push B off to a new process, but be able to program as if it 
> were still in the same process as A. That is, I'd like to be able to 
> call B's member functions from A without having to teach A how to do 
> interprocess communication.
> 
> The solution I've been thinking of is to write 'avatar' objects that 
> represent objects living in different processes and know how to pass 
> member function calls along to their 'true selves' (probably using 
> mpi4py and/or IPython1, but I haven't worked out the details yet). If A 
> lives in process 0 and B gets pushed to process 1, I would create an 
> avatar of B in process 0 and one of A in process 1.
> 
> That scheme would get kind of klunky, though. Has anyone thought about/ 
> dealt with this situation before?

Yes, lots. You almost certainly don't want to roll your own solution. Look at
Twisted's Perspective Broker (links near the bottom of the page):

  http://twistedmatrix.com/projects/core/documentation/howto/index.html

and Pyro:

  http://pyro.sourceforge.net/

There are a handful of other solutions, too.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco