advice on sub-classing multiprocessing.Process and multiprocessing.BaseManager

[matt.newville at gmail.com]

On Monday, March 24, 2014 7:19:56 PM UTC-5, Chris Angelico wrote:
> On Tue, Mar 25, 2014 at 7:24 AM, Matt Newville
> 
> > I'm maintaining a python interface to a C library for a distributed
> > control system (EPICS, sort of a SCADA system) that does a large
> > amount of relatively light-weight network I/O.   In order to keep many
> > connections open and responsive, and to provide a simple interface,
> > the python library keeps a global store of connection state.
> >
> > This works well for single processes and threads, but not so well for
> > multiprocessing, where the global state causes trouble.
> 
> 
> From the sound of things, a single process is probably what you want
> here. Is there something you can't handle with one process?

Thanks for the reply.  I find that appreciation is greatly (perhaps infinitely) delayed whenever I reply "X is probably not what you want to do" without further explanation to a question of "can I get some advice on how to do X?". So, I do thank you for your willingness to reply, even such a guaranteed-to-be-under-appreciated reply. 

There are indeed operations that can't be handled with a single process, such as simultaneously using multiple cores.  This is why we want to use multiprocessing instead of (or, in addition to) threading.  We're trying to do real-time collection of scientific data from a variety of data sources, generally within a LAN. The data can get largish and fast, and intermediate processing occasionally requires non-trivial computation time.  So being able to launch worker processes that can run independently on separate cores would be very helpful.  Ideally, we'd like to let sub-processes make calls to the control system too, say, read new data.

I wasn't really asking "is multiprocessing appropriate?" but whether there was a cleaner way to subclass multiprocessing.BaseManager() to use a subclass of Process().  I can believe the answer is No, but thought I'd ask.

Thanks again,

--Matt