advice on sub-classing multiprocessing.Process and multiprocessing.BaseManager

[matt.newville at gmail.com]

ChrisA -

>> I wasn't really asking "is multiprocessing appropriate?" but whether
>> there was a cleaner way to subclass multiprocessing.BaseManager() to 
>> use a subclass of Process().  I can believe the answer is No, but 
>> thought I'd ask.
> 
> I've never subclassed BaseManager like this. It might be simpler to
> spin off one or more workers and not have them do any network
> communication at all; that way, you don't need to worry about the
> cache. Set up a process tree with one at the top doing only networking
> and process management (so it's always fast), and then use a
> multiprocessing.Queue or somesuch to pass info to a subprocess and
> back. Then your global connection state is all stored within the top
> process, and none of the others need care about it. You might have a
> bit of extra effort to pass info back to the parent rather than simply
> writing it to the connection, but that's a common requirement in other
> areas (eg GUI handling - it's common to push all GUI manipulation onto
> the main thread), so it's a common enough model.
> 
> But if subclassing and tweaking is the easiest way, and if you don't
> mind your solution being potentially fragile (which subclassing like
> that is), then you could look into monkey-patching Process. Inject
> your code into it and then use the original. It's not perfect, but it
> may turn out easier than the "subclass everything" technique.
> 
> ChrisA

Thanks, I agree that restricting network communications to a parent process would be a good recommended solution, but it's hard to enforce and easy to forget such a recommendation.  It seems better to provide lightweight library-specific subclasses of Process (and Pool) and explaining why they should be used.  This library (pyepics) already does similar things for interaction with other libraries (notably providing decorators to avoid issues with wxPython). 

Monkey-patching multiprocessing.Process seems more fragile than subclassing it.  It turned out that multiprocessing.pool.Pool was also very easy to subclass.  But cleanly subclassing the Managers in multiprocessing.managers look much harder.  I'm not sure if this is intentional or not, or if it should be filed as an issue for multiprocessing.   For now, I'm willing to say that the multiprocessing managers are not yet available with the pyepics library.

Thanks again,

--Matt