[Web-SIG] Web Site Process Bus

[Robert Brewer]

Chris McDonough wrote:
> On Jun 26, 2007, at 2:39 PM, Robert Brewer wrote:
> 
> > Chris McDonough wrote:
> >> There are also non-webbish processes like postgres, mysql, 
> >> etc. that
> >> need to be treated as "part of the application".
> >>
> >> I handle this currently by running all of the processes 
> >> related to a
> >> specific project under a process controller (which happens to be
> >> implemented in Python, but that's besides the point, see http://
> >> www.plope.com/software/supervisor2/).  The process controller is
> >> responsible for execing the child processes upon its own startup.
> >> It is also responsible for restarting children if they die,
> >> capturing their output (if any), and allowing sufficiently
> >> privileged users to start and stop each one independently.
> >> The only promise a subprocess must make to be managed is that
> >> it must be possible to start the process "in the foreground"
> >> (not under its down daemon manager).
> >>
> >> If a "process bus" is implemented I suspect it should be 
> >> implemented at this kind of level.
> >
> > Ah, but there's the rub: we all have different ideas about how to
> > *implement* IPC and control.
> 
> I'm confused by this in your earlier message, describing example  
> scenarios:
> 
> """
> If I'm primarily a Zope user instead, I might start my website with
> zdaemon. This would work exactly like the above, but the Bus object
> would be instantiated and started by the zdaemon package. If I'm using
> Graham's new mod_wsgi with Apache, I might expect it to create and
> control the Bus.
> """
> 
> I think I'm mostly confused by the name "process bus" because it  
> seems like the primary use case for something like this is where all  
> of the applications share the same process space

I don't see why it should be limited by that. The primary use case is
anywhere site components and application components are interacting,
that could benefit from a shared understanding (and control) of the
state of the site. To me, that requires a common set of messages, but
the transport mechanism for those messages should be flexible so that
it's useful in both multithread and multiprocess architectures.

> ...and are all written in Python.  Am I right?

That's the initial target market, yes. But I think we can design the
messaging spec to be useful with non-Python application components.

> If so, maybe a different name is in order?   
> "Application Bus"?  Or even "WSGI Bus", if its presumed that all of  
> the applications will be WSGI applications?

Sure, "application bus" is fine, although it's just the other side of
the same coin: "applications" on one side, "site" on the other.

I wouldn't want this to be "WSGI Bus", simply because there's no benefit
for that relationship; the two specs should be useful independently of
each other. In particular, we should be able to design a site-messaging
bus which works for WSGI 1.0, 1.1, 2.0, and whatever might obsolete the
current WSGI in the future.

> I'm confused because zdaemon is a generic process controller, it  
> knows nothing in particular about the application running under it  
> except that it's a UNIX process. It could start postgres instead of  
> Zope if you configured it to.

Sorry, I was really thinking of zopectl when I wrote that. I'm not sure
how zdaemon itself would fit into this whole scheme--it's the process
which zdaemon invokes that should be directly involved. If that process
is a good provider of bus-aware services, you might not need zdaemon
anymore.

> If zdaemon creates a Bus object,
> nothing will be able to send messages to the bus except zdaemon  
> itself, and there can't be any useful listeners because it doesn't  
> share the same process space as its child.

If you use the example Bus implementation I posted, then yes. That's why
I'm pitching WSPBus as a spec, not an implementation. Multiprocess
controllers could implement the bus using any of various forms of IPC;
they just need to arrange for each application component to get a Bus
object that, behind the scenes, is specific to the chosen method of IPC.

So, yes, interprocess communication is more complicated than
intraprocess. But that's true whether you standardize on a bus spec or
not.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org