[Web-SIG] daemon tools

[Robert Brewer]

Jim Fulton wrote:
> On Mar 5, 2007, at 1:38 PM, Robert Brewer wrote:
> > ...where the "pywebd" module:
> >
> >  1. Composes the WSGI stack (provides a library to do so at least),
> >  2. Notifies frameworks of site-wide events (like start, 
> stop, restart
> > and graceful),
> >  3. Provides plugins that frameworks can "notify"; for 
> example, adding
> > files to an autoreload plugin.
> 
> This sounds great to me.

I wasn't expecting such quick agreement. ;)

For anyone's information, I've started developing just such a beast in
the CherryPy trunk:
http://www.cherrypy.org/browser/trunk/cherrypy/pywebd

CherryPy will probably continue to distribute it as a subpackage just
for ease of install, but it won't have any CP dependencies. If others
are really interested in developing this collaboratively, I'd be happy
to make it its own project and solicit committers. In particular,
there's no "webctl" module yet (because we need more discussion on its
role before I commit to a direction).

> I see a division of responsibilities between:
> 
> * A facility for managing an application process
> 
>    - start/stop/status/etc
> 
>    - passing environment variables, providing some logging 
>      support if necessary (especially for applications that
>      spew to standard err/out).
> 
>    - Optionally providing other daemon behaviors like
>      disconnecting from the controlling terminal, changing
>      user, etc.  zdaemon provides this service on behalf of
>      applications.
> 
> * A main program that provides common application-level 
> services like the ones you describe above.
> 
>    - Optionally providing other daemon behaviors like
>      disconnecting from the controlling terminal, changing
>      user, etc.  ll.daemon provides some of these services
>      within an application.
> 
> A question is whether to provide the daemonizing support in the main  
> program or in the controlling program.

The "main program" should have the daemonization support. This would
allow framework authors to continue providing "quickstart" and stop
calls to their users as a full-featured alternative to invoking the
controlling program (where "full-featured" includes daemonization,
etcetera). IMO the controlling program ("webctl") wouldn't do any of
your "optional daemon behaviors"; instead, it would be a command-line
way to specify/collect an environment (including config files), start
the main program, and then asynchronously send messages to the main
program like "stop" and "status". It would run, execute a command, and
then exit (much like apachectl does).

This is also pretty much how I see zdctl operating, with a few areas I'd
like to investigate:

 1. I would very much like webctl to be the component that understands a
WSGI-composition config format or formats. Or rather, I don't want
pywebd to fuss with that--pywebd should understand the entry points and
use/expose an API for composing a WSGI stack, but that should be an
imperative API, so that frameworks can do their own composition for the
user. For example, TG silently adds URL handlers for Mochikit (that
shouldn't have to be included in a config file by the user).
 2. AF_UNIX isn't available on Windows. I'd like to find ways of passing
status back from pywebd to webctl that don't involve a socket.
 3. zdctl spawns zdrun (right?). I'd like webctl to spawn pywebd, but
currently I'm calling the whole package "pywebd". I probably need to
change:

/pywebd
    __init__.py
    base.py
    plugins.py
    win32.py

...to a more separated arrangement:

/pyweb (other name ideas most welcome)
    __init__.py
    base.py
    plugins.py
    pywebd(.exe)
    unix.py
    webctl(.exe)
    win32.py

> Note that in answering this question, we probably need to have an
> idea how this will work on windows.  If Unix-specific daemonizing
> code is in the main application, then the application won't be
> portable. Of course, if the main program is generic, it might not
> be a big deal to have separate versions for Windows and Unix.

My hope is that pywebd will have a "win32" module (as my initial foray
does). Perhaps I should move the daemonization plugin to a "unix"
(posix?) module.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org