From graham.dumpleton at gmail.com  Sun Jun  3 08:06:21 2007
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Sun, 3 Jun 2007 16:06:21 +1000
Subject: [Web-SIG] WSGI application object returning a string/buffer.
Message-ID: <88e286470706022306t30cad62q8056d93c1cf9dce2@mail.gmail.com>

The WSGI specification says:

"""When called by the server, the application object must return an
iterable yielding zero or more strings. This can be accomplished in a
variety of ways, such as by returning a list of strings, or by the
application being a generator function that yields strings, or by the
application being a class whose instances are iterable. Regardless of
how it is accomplished, the application object must always return an
iterable yielding zero or more strings."""

This requirement would be satisfied by an application object returning
the response content using:

  # list of strings
  return ['s1','s2']

  # tuple of strings
  return ('s1','s2')

  # generator
  yield 's1'
  yield 's2'

One could also return any object which is iterable, provide it yields
strings for each item.

A consequence of this last option though is that the following are
actually valid:

  # string
  return 's1s2'

  # buffer
  return buffer('s1s2')

This is because when iterated over, the result is a string object for
each item albeit where each string object is of length 1.

Although string and buffer objects can be returned, one cannot however
return a unicode object. Ie., the following will fail:

  # unicode (fails)
  return u's1s2'

This fails because each item yield when iterated over is a unicode object.

For the string and buffer return value, although it is technically
legal by the specification, that it is also a requirement that data be
flushed between each yielded item from a sequence means that each
character is written back to the user one at a time. The result of
this is that performance will suck something terrible.

The question is, although returning a string object is technically
allowed by the specification, should a good WSGI adapter actually
allow it given that it is more than likely a programming mistake that
someone has done it this way rather than return the string object
enclosed in an array?

The buffer object is a more problematic case, as they can't even be
returned in an array as is, but would need to be converted to a string
object first before being put in the array.

Any comments or suggestion? Should I just process string and buffer
objects as a sequence without complaint as required by specification,
or disallow it based on probability that it isn't probably what the
user meant to do anyway?

Graham

From pje at telecommunity.com  Sun Jun  3 21:30:44 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 03 Jun 2007 15:30:44 -0400
Subject: [Web-SIG] WSGI application object returning a  string/buffer.
In-Reply-To: <88e286470706022306t30cad62q8056d93c1cf9dce2@mail.gmail.com
 >
References: <88e286470706022306t30cad62q8056d93c1cf9dce2@mail.gmail.com>
Message-ID: <20070603193046.B45583A40AC@sparrow.telecommunity.com>

At 04:06 PM 6/3/2007 +1000, Graham Dumpleton wrote:
>Should I just process string and buffer
>objects as a sequence without complaint as required by specification,

I'd say yes.  It's trivial for someone to use wsgiref.validate to 
test their application, and it will tell them about this issue.

You could, of course, provide an option to use wsgiref.validate 
automatically...


From graham.dumpleton at gmail.com  Mon Jun  4 00:32:21 2007
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Mon, 4 Jun 2007 08:32:21 +1000
Subject: [Web-SIG] WSGI application object returning a string/buffer.
In-Reply-To: <20070603193046.B45583A40AC@sparrow.telecommunity.com>
References: <88e286470706022306t30cad62q8056d93c1cf9dce2@mail.gmail.com>
	<20070603193046.B45583A40AC@sparrow.telecommunity.com>
Message-ID: <88e286470706031532nba823cs851acdaa4ccf9955@mail.gmail.com>

On 04/06/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 04:06 PM 6/3/2007 +1000, Graham Dumpleton wrote:
> >Should I just process string and buffer
> >objects as a sequence without complaint as required by specification,
>
> I'd say yes.  It's trivial for someone to use wsgiref.validate to
> test their application, and it will tell them about this issue.

Last time I looked the wsgiref.validate function will only complain
about string being returned, not 'buffer'.

> You could, of course, provide an option to use wsgiref.validate
> automatically...

It is trivial for someone to do it themselves in their WSGI
application script file and a lot harder to add such an option as in
inbuilt feature, plus would add a dependency on some third party
package.

Graham

From sasch.pe at gmx.de  Sat Jun 23 11:37:07 2007
From: sasch.pe at gmx.de (Sascha Peilicke)
Date: Sat, 23 Jun 2007 11:37:07 +0200
Subject: [Web-SIG] Clientside Python
Message-ID: <1182591427.15675.29.camel@schlepp>

Hello,

Hello, hopefully this is the right place, I would like to propose a new
application area for Python. But lets start from the beginning. Python
already started to conquer the server-side via all those cool frameworks
(Django,Pylons,TurboGears,...), basically everything related to networks
is available. But one thing is missing: client-side python-scripting in
the web-browser! Think about it, there are basically only two languages
usable for this purpose, VBScript, no option. JavaScript, the status quo
but it is more a thing one has to work with and not want to work with.
Like these guys from the MochiKit framework " ... make JavaScript suck
less" and it really sucks! Wouldn't it be cool to have our favorite
language at hand when developing a web-app? 
I am not sure about how difficult this task could be. The interpreter
runs on every platform which can run a browser, it can be embedded in
basically every kind of software. The thing to do would be creating a
fully-compatible DOM-Implementation, XMLHttpRequest and some kind of
wrapping middleware which acts as the interface for the specific
browser. I tried to do some programmer's art:

-----------   --------------  ------------------------
| Browser |   | Middleware |  | Python-Interpreter   |
|         |===| (glue code)|==| DOM-Stack            |
-----------   --------------  | Reduced Standard-Lib |
                              | XMLHttpRequest       |
                              | ...                  |
                              ------------------------

Then the following will become possible then:

...
<script type="text/python">
   def compute():
   	2*3
</script>
...
<div onload="compute"></div>
...

Altough I see a slight problem: correct indentation, which is important
for python code inside <script> tags. I am sure this could be solved
somehow. Another point: License, we can't say just GPL or free licenses
in general, to become successfull it has to work in proprietary
browsers, a thing to talk about. As a last point we could implement a
unique feature, let me explain that point. Everybody knows the
XMLHttpRequest object, which made Web2.0 possible. It is cool to do
asynchronous request but only for the download direction. What about an
object which does the same for upload. Example: I want to upload a big
pdf file in my plone site. This usually takes time, lots of time. It is
possible to do this asynchronously with nasty JavaScript hacks
(toghether with an iframe..). I call it XMLHttpResponse :) 


I am looking foward to hearing from you, all the best,
Sascha Peilicke
-- 
http://saschashideout.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.python.org/pipermail/web-sig/attachments/20070623/42d454e8/attachment.pgp 

From robinbryce at gmail.com  Sat Jun 23 12:21:58 2007
From: robinbryce at gmail.com (Robin Bryce)
Date: Sat, 23 Jun 2007 11:21:58 +0100
Subject: [Web-SIG] Clientside Python
In-Reply-To: <1182591427.15675.29.camel@schlepp>
References: <1182591427.15675.29.camel@schlepp>
Message-ID: <bcf87d920706230321h539a320hddcdf810d5685f11@mail.gmail.com>

> is available. But one thing is missing: client-side python-scripting in
> the web-browser!

You might be interested in Brett Cannons work see:

http://mail.python.org/pipermail/python-dev/2006-June/066344.html

AIUI: The restricted execution issue needs to be addressed before
anything else can happen along the lines you suggest.

On the other hand its going to be very interesting to see the impact
this has on browser scripting http://www.mozilla.org/projects/tamarin/

The browser / http platform is a _very_ big tent and python is just
one player. In the long run I (wildly) speculate that techniques like
those employed by PyPy (which has a js backend ISTR) will make the
question of which browser scripting language is supported a moot
point.

Interfaces & standards are a different issue though.

> fully-compatible DOM-Implementation, XMLHttpRequest and some kind of
> wrapping middleware which acts as the interface for the specific
> browser.

DOM - surely there is one around somewhere ?

XMLHttpRequest - writing a library that wraps urllib2/httplib (from
the standard lib) could be a good place to start.

> object which does the same for upload. Example: I want to upload a big
> pdf file in my plone site. This usually takes time, lots of time. It is
> possible to do this asynchronously with nasty JavaScript hacks

nasty JavaScript hacks ? Pushing or streaming data from the client
side using XMLHttpRequest is *not* a java script hack - it's a http
hack ;-)

Have fun :-)

Robin

On 23/06/07, Sascha Peilicke <sasch.pe at gmx.de> wrote:
> Hello,
>
> Hello, hopefully this is the right place, I would like to propose a new
> application area for Python. But lets start from the beginning. Python
> already started to conquer the server-side via all those cool frameworks
> (Django,Pylons,TurboGears,...), basically everything related to networks
> is available. But one thing is missing: client-side python-scripting in
> the web-browser! Think about it, there are basically only two languages
> usable for this purpose, VBScript, no option. JavaScript, the status quo
> but it is more a thing one has to work with and not want to work with.
> Like these guys from the MochiKit framework " ... make JavaScript suck
> less" and it really sucks! Wouldn't it be cool to have our favorite
> language at hand when developing a web-app?
> I am not sure about how difficult this task could be. The interpreter
> runs on every platform which can run a browser, it can be embedded in
> basically every kind of software. The thing to do would be creating a
> fully-compatible DOM-Implementation, XMLHttpRequest and some kind of
> wrapping middleware which acts as the interface for the specific
> browser. I tried to do some programmer's art:
>
> -----------   --------------  ------------------------
> | Browser |   | Middleware |  | Python-Interpreter   |
> |         |===| (glue code)|==| DOM-Stack            |
> -----------   --------------  | Reduced Standard-Lib |
>                               | XMLHttpRequest       |
>                               | ...                  |
>                               ------------------------
>
> Then the following will become possible then:
>
> ...
> <script type="text/python">
>    def compute():
>         2*3
> </script>
> ...
> <div onload="compute"></div>
> ...
>
> Altough I see a slight problem: correct indentation, which is important
> for python code inside <script> tags. I am sure this could be solved
> somehow. Another point: License, we can't say just GPL or free licenses
> in general, to become successfull it has to work in proprietary
> browsers, a thing to talk about. As a last point we could implement a
> unique feature, let me explain that point. Everybody knows the
> XMLHttpRequest object, which made Web2.0 possible. It is cool to do
> asynchronous request but only for the download direction. What about an
> object which does the same for upload. Example: I want to upload a big
> pdf file in my plone site. This usually takes time, lots of time. It is
> possible to do this asynchronously with nasty JavaScript hacks
> (toghether with an iframe..). I call it XMLHttpResponse :)
>
>
> I am looking foward to hearing from you, all the best,
> Sascha Peilicke
> --
> http://saschashideout.de
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/robinbryce%40gmail.com
>
>
>

From johnny at debris.demon.nl  Sat Jun 23 17:15:50 2007
From: johnny at debris.demon.nl (Johnny deBris)
Date: Sat, 23 Jun 2007 17:15:50 +0200
Subject: [Web-SIG] Clientside Python
In-Reply-To: <1182591427.15675.29.camel@schlepp>
References: <1182591427.15675.29.camel@schlepp>
Message-ID: <467D3926.6040400@debris.demon.nl>

Sascha Peilicke wrote:
> 
> ... Wouldn't it be cool to have our favorite
> language at hand when developing a web-app? 

If you're don't mind having a bit of a different model, PyPy might be
interesting for you. They have a (relatively complete) Python to
JavaScript convertor, which allows you to write and test your code in
RPython, a subset of Python that's a little less dynamic (so not Python
syntax with JS semantics or anything), and when done convert it to
JavaScript to execute in the browser.

Although the feature is not as mature as some other parts of PyPy, and
even though RPython is by far not as nice to write and debug as Python
is, it's not a bad platform to work on (well, not as bad as JS is
generally percieved at least ;) and there are some nice additional
tricks provided too (such as a simple RPC thingie over
XMLHttpRequest/JSON). Also, there are DOM and XMLHttpRequest wrappers to
allow testing client-side code in a plain interpreter, which might be
useful for you to base your code on if you decide to give it a go...

For more information, see
http://codespeak.net/pypy/dist/pypy/doc/js/using.html

By the way, there once was an Active-X plugin for Python too, but iirc
that was cancelled because it's very hard, if not impossible, to sandbox
the interpreter properly...

Cheers,

Guido Wesdorp

From fumanchu at amor.org  Mon Jun 25 19:57:11 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 10:57:11 -0700
Subject: [Web-SIG] Web Site Process Bus
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>

I'd like to continue talking about standardization on site-wide process
state and services.

As described more fully on my blog [1], I'm proposing we create a new
spec for a simple publish-subscribe Bus to manage site-wide state
transitions (start, stop, etc.) and connect application components with
site-wide services (daemonize, autoreload, site logging, etc).

This would NOT be a daemon, nor would it introduce any dependencies. It
would simply say that the main program (be it a framework entry point,
or Apache, or a user script) creates a Bus object according to the spec,
and then gives a reference to that Bus object to all participating
components that care about it.

The blog post might seem like I have a finished candidate for this; far
from it. This should be a collaborative effort, and I'm very open to
discussion at all levels of detail. Even if this flies at the highest
conceptual level, there are still several things I know of we would need
to nail down:

 * States. For example, do we need an EXITING state? If so, we should
probably rename STOPPED to IDLE.
 * State transitions: do we need to lock or block which Bus methods can
be called, depending on the current state?
 * Standard non-state channels, such as signals. Do we want a 'status'
channel (for e.g. zdaemon)?
 * Should bus.log take other args, like 'severity'? Should it assume a
stdlib logging API?
 * Should bus.log be replaced with separate stdout and stderr instead?
 * ...and just to introduce a bikeshed, is there a better name for the
spec effort than "WSPB"?

I have lots of other questions, but I'll let you all ask them.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

[1]
http://www.aminus.org/blogs/index.php/fumanchu/2007/06/24/web_site_proce
ss_bus

From pje at telecommunity.com  Mon Jun 25 20:50:52 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 25 Jun 2007 14:50:52 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchang
	e.local>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
Message-ID: <20070625184846.EDC683A409B@sparrow.telecommunity.com>

At 10:57 AM 6/25/2007 -0700, Robert Brewer wrote:
>This should be a collaborative effort, and I'm very open to
>discussion at all levels of detail. Even if this flies at the highest
>conceptual level, there are still several things I know of we would need
>to nail down:
>
>  * States. For example, do we need an EXITING state? If so, we should
>probably rename STOPPED to IDLE.
>  * State transitions: do we need to lock or block which Bus methods can
>be called, depending on the current state?
>  * Standard non-state channels, such as signals. Do we want a 'status'
>channel (for e.g. zdaemon)?
>  * Should bus.log take other args, like 'severity'? Should it assume a
>stdlib logging API?
>  * Should bus.log be replaced with separate stdout and stderr instead?
>  * ...and just to introduce a bikeshed, is there a better name for the
>spec effort than "WSPB"?
>
>I have lots of other questions, but I'll let you all ask them.

I have some more basic questions...  like, how do you know you need 
to subscribe in the first place?  Who does the subscribing?

My initial reaction is that I like the general idea, but the idea of 
having a bus *object* seems wrong to me somehow.  Perhaps because the 
only way to get it seems to be in an already-running application, 
which seems to sort of defeat the purpose.  But maybe my purposes are 
different than yours?

Perhaps if you could present some example scenarios to show how you 
see this actually being used...?


From ianb at colorstudy.com  Mon Jun 25 21:28:22 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 25 Jun 2007 14:28:22 -0500
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625184846.EDC683A409B@sparrow.telecommunity.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<20070625184846.EDC683A409B@sparrow.telecommunity.com>
Message-ID: <46801756.4080709@colorstudy.com>

Phillip J. Eby wrote:
> At 10:57 AM 6/25/2007 -0700, Robert Brewer wrote:
>> This should be a collaborative effort, and I'm very open to
>> discussion at all levels of detail. Even if this flies at the highest
>> conceptual level, there are still several things I know of we would need
>> to nail down:
>>
>>  * States. For example, do we need an EXITING state? If so, we should
>> probably rename STOPPED to IDLE.
>>  * State transitions: do we need to lock or block which Bus methods can
>> be called, depending on the current state?
>>  * Standard non-state channels, such as signals. Do we want a 'status'
>> channel (for e.g. zdaemon)?
>>  * Should bus.log take other args, like 'severity'? Should it assume a
>> stdlib logging API?
>>  * Should bus.log be replaced with separate stdout and stderr instead?
>>  * ...and just to introduce a bikeshed, is there a better name for the
>> spec effort than "WSPB"?
>>
>> I have lots of other questions, but I'll let you all ask them.
> 
> I have some more basic questions...  like, how do you know you need 
> to subscribe in the first place?  Who does the subscribing?
> 
> My initial reaction is that I like the general idea, but the idea of 
> having a bus *object* seems wrong to me somehow.  Perhaps because the 
> only way to get it seems to be in an already-running application, 
> which seems to sort of defeat the purpose.  But maybe my purposes are 
> different than yours?
> 
> Perhaps if you could present some example scenarios to show how you 
> see this actually being used...?

I'm also confused about what this is about.  Or maybe it feels like it's 
about more than one thing, and needn't be.  The blog post kind of helps 
and also kind of confuses things.

For instance, doesn't atexit work fine for exiting?  (Not in mod_python, 
but I still don't know why it *shouldn't* work in mod_python.)  Of 
course, there isn't anything for other signals, and creating something 
for those would be quite useful.  Potentially a Zope-style minimal event 
framework would work.  Maybe something like:

   send_signal(signal_name, signal_data)
   subscribe(signal_name, listener)

Where in the second case, listener(signal_name, signal_data) is called, 
and signal_name can be used to automatically filter, or None to receive 
all signals.  I prefer strings for the names, because then there's no 
dependencies (you don't have to import anything, you just have to have 
packages agree on names).  I think Zope doesn't have any high-level 
filtering (?) and generally uses interfaces for filtering in the 
listener itself.

Anyway, given that we still would need to define the signals we care 
about in the web context, but it's not too terrible hard I think.

Maybe the "bus" you are talking about that is registered is the 
thing-that-sends-signals?  I think it can simply be a single 
module/library that we all agree to use.

Anyway, this mostly seems like what you are describing.  I guess when 
viewed as an event publishing system, there's other concerns that pop up 
-- can you cancel the event?  (Ugh, I don't think so; I think GUI events 
and signals are something fairly different from what we're talking about 
here).  Also, should exceptions propagate?  I feel vaguely like they 
should, but I'm not sure.

-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
             | Write code, do good | http://topp.openplans.org/careers

From benji at benjiyork.com  Mon Jun 25 21:43:19 2007
From: benji at benjiyork.com (Benji York)
Date: Mon, 25 Jun 2007 15:43:19 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
Message-ID: <46801AD7.8010908@benjiyork.com>

Robert Brewer wrote:
> I'd like to continue talking about standardization on site-wide process
> state and services.

Perhaps zope.event would be a good fit. 
http://svn.zope.org/zope.event/trunk/src/zope/event/README.txt?view=markup
-- 
Benji York
http://benjiyork.com

From pje at telecommunity.com  Mon Jun 25 22:18:05 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 25 Jun 2007 16:18:05 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <46801756.4080709@colorstudy.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<20070625184846.EDC683A409B@sparrow.telecommunity.com>
	<46801756.4080709@colorstudy.com>
Message-ID: <20070625201558.31A3C3A409B@sparrow.telecommunity.com>

At 02:28 PM 6/25/2007 -0500, Ian Bicking wrote:
>Potentially a Zope-style minimal event framework would work.  Maybe 
>something like:
>
>   send_signal(signal_name, signal_data)
>   subscribe(signal_name, listener)

That was what I was wondering, too, except I was thinking it would be 
sufficient to use entry points for subscription, but only invoke the 
entry points whose modules are in sys.modules.  In other words, never 
actually import a module in order to invoke a callback.  That way, 
subscription is a natural side effect of importing the modules that 
contain the listeners.  Something like:

def send_signal(group, name, *args, **kw):
     for ep in iter_entry_points(group, name):
         if ep.module_name in sys.modules:
             ep.load()(*args, **kw)

would then be sufficient.  pkg_resources is currently proposed for 
inclusion in Python 2.6 (per PEP 365), so there wouldn't even be any 
non-stdlib requirements then.  And of course the implementation is 
trivial enough to copy into any code that wants to send signals.

Of course, if you want prioritized callbacks, you can always sort the 
ep.load() results using a priority attribute or some such.  I'm not 
sure priority makes much sense, though, due to the difficulty of 
determining a global priority scheme.


From fumanchu at amor.org  Mon Jun 25 22:43:45 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 13:43:45 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625184846.EDC683A409B@sparrow.telecommunity.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DAFC@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> At 10:57 AM 6/25/2007 -0700, Robert Brewer wrote:
> >This should be a collaborative effort, and I'm very open to
> >discussion at all levels of detail. Even if this flies at the highest
> >conceptual level, there are still several things I know of 
> >we would need to nail down:
> >
> >  * States. For example, do we need an EXITING state? If so, 
> >we should probably rename STOPPED to IDLE.
> >  * State transitions: do we need to lock or block which Bus 
> >methods can be called, depending on the current state?
> >  * Standard non-state channels, such as signals. Do we want 
> >a 'status' channel (for e.g. zdaemon)?
> >  * Should bus.log take other args, like 'severity'? Should 
> >it assume a stdlib logging API?
> >  * Should bus.log be replaced with separate stdout and 
> >stderr instead?
> >  * ...and just to introduce a bikeshed, is there a better 
> >name for the spec effort than "WSPB"?
> >
> >I have lots of other questions, but I'll let you all ask them.
> 
> I have some more basic questions...  like, how do you know you need 
> to subscribe in the first place?  Who does the subscribing?
> 
> My initial reaction is that I like the general idea, but the idea of 
> having a bus *object* seems wrong to me somehow.  Perhaps because the 
> only way to get it seems to be in an already-running application, 
> which seems to sort of defeat the purpose.  But maybe my purposes are 
> different than yours?
> 
> Perhaps if you could present some example scenarios to show how you 
> see this actually being used...?

Sure. Let's assume I want to have a single host+port serve apps from two
different WSPB-enabled frameworks; let's use hypothetical future
versions of Pylons and Zope.

If I'm primarily a Pylons user, I'm used to starting my websites with
"paster serve". In this case, paste.script would create a WSPBus object.
[It's up to the Paste developers whether to distribute their own
wspbus.py module, or to require one via setuptools.] When Paste parses
the paste.deploy file and composes the WSGI stack, it somehow hands a
Bus reference to Pylons. Exactly how is up for debate; Pylons might
provide a hook for it (say, in app_globals.py), or Pylons components
might take the bus as a constructor arg.

Paste's ServeCommand object would load the WSGI components and server
(as it does now), but then, instead of calling server(app), it would
call bus.subscribe('start', server, app). Pylons and Zope would also
have the opportunity to subscribe listeners to the 'start' channel (and
other channels), such as database connections, access log files, etc.
Then, instead of calling server(app) directly, Paste would call
bus.start(). It could then call bus.block(states=states.STOPPED) instead
of trapping KeyboardInterrupt itself. If block() traps a KBInt or
SyExit, it sends the 'stop' message to all subscribers, regardless of
whether they were created by Paste, Pylons, or Zope. They all shut down
synchronously, and then Paste could feel much safer about calling
bus.exit(). If KBInt or SysExit are trapped by a Pylons or Zope
component, they would be expected to call bus.stop or bus.exit (which
would unblock Paste).

If I'm primarily a Zope user instead, I might start my website with
zdaemon. This would work exactly like the above, but the Bus object
would be instantiated and started by the zdaemon package. If I'm using
Graham's new mod_wsgi with Apache, I might expect it to create and
control the Bus.

If I'm using paste.script but I want autoreload to use execv instead of
the default child-process style of autoreload, this is much easier with
the WSPBus. Instead of hacking paste.script, I unsubscribe Paste's
default autoreloader from the 'restart' channel and subscribe my own.

I'll stop there for now.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From fumanchu at amor.org  Mon Jun 25 22:51:09 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 13:51:09 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <46801AD7.8010908@benjiyork.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchange.local>

Benji York wrote:
> Robert Brewer wrote:
> > I'd like to continue talking about standardization on 
> site-wide process
> > state and services.
> 
> Perhaps zope.event would be a good fit. 
> http://svn.zope.org/zope.event/trunk/src/zope/event/README.txt
> ?view=markup

All 4 lines of it? ;)

    subscribers = []
    
    def notify(event):
        for subscriber in subscribers:
            subscriber(event)

Seriously, though, this handles the notification but not the state
machine, which I think is critical to the effort. It also doesn't do any
error-handling for misbehaving subscribers, so not all subscribers are
guaranteed to run if there's an unhandled error in an earlier
subscriber. It also sends all events to all subscribers, instead of
partitioning the subscribers. And finally, I don't want "import
zope.event" in my framework.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From pje at telecommunity.com  Mon Jun 25 23:08:52 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 25 Jun 2007 17:08:52 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchang
	e.local>
References: <46801AD7.8010908@benjiyork.com>
	<435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchange.local>
Message-ID: <20070625210646.7A0F23A409B@sparrow.telecommunity.com>

At 01:51 PM 6/25/2007 -0700, Robert Brewer wrote:
>Seriously, though, this handles the notification but not the state
>machine, which I think is critical to the effort. It also doesn't do any
>error-handling for misbehaving subscribers, so not all subscribers are
>guaranteed to run if there's an unhandled error in an earlier
>subscriber.

Again, it would be really helpful if you could provide some scenarios 
that show why these things are important.  It's not immediately 
obvious to me that they are.

For example, if an error occurs, isn't that an indication that the 
component is broken?  Masking the failure just makes it less likely 
the component will get fixed in a timely manner.

Meanwhile, if you get a start call, you must be starting, right?  So 
why worry about the state?  It'd be simpler to just use 
"before/during/after" messages the way Twisted does.  Your "block" 
example could be replaced by waiting for the "after" message of the 
desired state, for example.


From benji at benjiyork.com  Mon Jun 25 23:06:46 2007
From: benji at benjiyork.com (Benji York)
Date: Mon, 25 Jun 2007 17:06:46 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchange.local>
Message-ID: <46802E66.9090703@benjiyork.com>

Robert Brewer wrote:
> All 4 lines of it? ;)

A thing of beauty, isn't it. <wink>

In all seriousness, it's served Zope 3 well for quite some time.
-- 
Benji York
http://benjiyork.com

From fumanchu at amor.org  Mon Jun 25 23:17:14 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 14:17:14 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625210646.7A0F23A409B@sparrow.telecommunity.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DBE6@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> At 01:51 PM 6/25/2007 -0700, Robert Brewer wrote:
> >Seriously, though, this handles the notification but not the state
> >machine, which I think is critical to the effort. It also 
> doesn't do any
> >error-handling for misbehaving subscribers, so not all 
> subscribers are
> >guaranteed to run if there's an unhandled error in an earlier
> >subscriber.
> 
> Again, it would be really helpful if you could provide some scenarios 
> that show why these things are important.  It's not immediately 
> obvious to me that they are.

I replied with some in another email, it's just being slow.

> For example, if an error occurs, isn't that an indication that the 
> component is broken?  Masking the failure just makes it less likely 
> the component will get fixed in a timely manner.

Yes, the component is broken. However, at runtime, breakage in a
CherryPy component shouldn't keep a Quixote component from, say,
correctly freeing its DB connections. The Bus implementation I posted
logs every failure, and then raises the last one. If we need more
notification than a site log, I think that can be added, either with an
optional argument to bus.log(), or with a custom subscriber to the 'log'
channel.

> Meanwhile, if you get a start call, you must be starting, right?  So 
> why worry about the state?  It'd be simpler to just use 
> "before/during/after" messages the way Twisted does.  Your "block" 
> example could be replaced by waiting for the "after" message of the 
> desired state, for example.

That's a possible way to go. My intention was to support both 1)
examination of the state by external components (for operations other
than 'block'--progress meters spring to mind) and 2) restrict some state
transitions if necessary; for example, make bus.start() do nothing (or
block) unless the state is "STOPPED".


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From jim at zope.com  Mon Jun 25 23:41:06 2007
From: jim at zope.com (Jim Fulton)
Date: Mon, 25 Jun 2007 17:41:06 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchange.local>
Message-ID: <F58D24E1-1132-40D7-9111-B7260905A64A@zope.com>


On Jun 25, 2007, at 4:51 PM, Robert Brewer wrote:

> Benji York wrote:
>> Robert Brewer wrote:
>>> I'd like to continue talking about standardization on
>> site-wide process
>>> state and services.
>>
>> Perhaps zope.event would be a good fit.
>> http://svn.zope.org/zope.event/trunk/src/zope/event/README.txt
>> ?view=markup
>
> All 4 lines of it? ;)
>
>     subscribers = []
>
>     def notify(event):
>         for subscriber in subscribers:
>             subscriber(event)
>
> Seriously, though, this handles the notification but not the state
> machine, which I think is critical to the effort. It also doesn't  
> do any
> error-handling for misbehaving subscribers, so not all subscribers are
> guaranteed to run if there's an unhandled error in an earlier
> subscriber. It also sends all events to all subscribers, instead of
> partitioning the subscribers.

Right.  You need a dispatching layer behind this.  The beauty of  
zope.event is that it puts very little requirement on people who  
publish events.

> And finally, I don't want "import
> zope.event" in my framework.

Why?  I have no a-priori objection to importing something that has  
cherrypy in its name.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From ianb at colorstudy.com  Mon Jun 25 23:42:16 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 25 Jun 2007 16:42:16 -0500
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625201558.31A3C3A409B@sparrow.telecommunity.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<20070625184846.EDC683A409B@sparrow.telecommunity.com>
	<46801756.4080709@colorstudy.com>
	<20070625201558.31A3C3A409B@sparrow.telecommunity.com>
Message-ID: <468036B8.5060105@colorstudy.com>

Phillip J. Eby wrote:
> At 02:28 PM 6/25/2007 -0500, Ian Bicking wrote:
>> Potentially a Zope-style minimal event framework would work.  Maybe 
>> something like:
>>
>>   send_signal(signal_name, signal_data)
>>   subscribe(signal_name, listener)
> 
> That was what I was wondering, too, except I was thinking it would be 
> sufficient to use entry points for subscription, but only invoke the 
> entry points whose modules are in sys.modules.  In other words, never 
> actually import a module in order to invoke a callback.  That way, 
> subscription is a natural side effect of importing the modules that 
> contain the listeners.  Something like:
> 
> def send_signal(group, name, *args, **kw):
>     for ep in iter_entry_points(group, name):
>         if ep.module_name in sys.modules:
>             ep.load()(*args, **kw)

I don't think that makes sense for this case.  The way I imagine using 
it is:

class MyApplication(object):

     def __init__(self, db):
         self.db = db
         subscribe('reload_resources', self.reload)
     def reload(self, data=None):
         self.db.close()
         self.db.open()

That is, I subscribe one particular thing when it is applicable, not 
because the library is on the system.


-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
             | Write code, do good | http://topp.openplans.org/careers

From fumanchu at amor.org  Mon Jun 25 23:41:51 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 14:41:51 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625210646.7A0F23A409B@sparrow.telecommunity.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DC80@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> Meanwhile, if you get a start call, you must be starting, right?  So 
> why worry about the state?  It'd be simpler to just use 
> "before/during/after" messages the way Twisted does.  Your "block" 
> example could be replaced by waiting for the "after" message of the 
> desired state, for example.

I just realized I haven't really explained what "start" and "stop" mean.
I think you might expect it to mean "beginning of process" and "exit the
process". But instead, I'm envisioning a FSM that has an "idle" state
in-between process init/exit and "server" start/stop, so that, without
restarting the process, you can stop and restart (un/bind the socket,
etc) the server components. This should also facilitate a daemon parent
process having a single site Bus and starting/stopping child processes
that contain the WSGI app and server components, if that's the way you
want to compose your site.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From ianb at colorstudy.com  Mon Jun 25 23:46:48 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 25 Jun 2007 16:46:48 -0500
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <468036B8.5060105@colorstudy.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>	<20070625184846.EDC683A409B@sparrow.telecommunity.com>	<46801756.4080709@colorstudy.com>	<20070625201558.31A3C3A409B@sparrow.telecommunity.com>
	<468036B8.5060105@colorstudy.com>
Message-ID: <468037C8.2050407@colorstudy.com>

Ian Bicking wrote:
> Phillip J. Eby wrote:
>> At 02:28 PM 6/25/2007 -0500, Ian Bicking wrote:
>>> Potentially a Zope-style minimal event framework would work.  Maybe 
>>> something like:
>>>
>>>   send_signal(signal_name, signal_data)
>>>   subscribe(signal_name, listener)
>> That was what I was wondering, too, except I was thinking it would be 
>> sufficient to use entry points for subscription, but only invoke the 
>> entry points whose modules are in sys.modules.  In other words, never 
>> actually import a module in order to invoke a callback.  That way, 
>> subscription is a natural side effect of importing the modules that 
>> contain the listeners.  Something like:
>>
>> def send_signal(group, name, *args, **kw):
>>     for ep in iter_entry_points(group, name):
>>         if ep.module_name in sys.modules:
>>             ep.load()(*args, **kw)
> 
> I don't think that makes sense for this case.  The way I imagine using 
> it is:
> 
> class MyApplication(object):
> 
>      def __init__(self, db):
>          self.db = db
>          subscribe('reload_resources', self.reload)
>      def reload(self, data=None):
>          self.db.close()
>          self.db.open()
> 
> That is, I subscribe one particular thing when it is applicable, not 
> because the library is on the system.

I forgot to add: this makes me think that some sort of weakref handling 
would be nice, because in this case I won't want the application to 
persist just because it's listening to this signal.

-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
             | Write code, do good | http://topp.openplans.org/careers

From ianb at colorstudy.com  Mon Jun 25 23:48:34 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 25 Jun 2007 16:48:34 -0500
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <F58D24E1-1132-40D7-9111-B7260905A64A@zope.com>
References: <435DF58A933BA74397B42CDEB8145A860D37DB28@ex9.hostedexchange.local>
	<F58D24E1-1132-40D7-9111-B7260905A64A@zope.com>
Message-ID: <46803832.6070205@colorstudy.com>

Jim Fulton wrote:
>> All 4 lines of it? ;)
>>
>>     subscribers = []
>>
>>     def notify(event):
>>         for subscriber in subscribers:
>>             subscriber(event)
>>
>> Seriously, though, this handles the notification but not the state
>> machine, which I think is critical to the effort. It also doesn't  
>> do any
>> error-handling for misbehaving subscribers, so not all subscribers are
>> guaranteed to run if there's an unhandled error in an earlier
>> subscriber. It also sends all events to all subscribers, instead of
>> partitioning the subscribers.
> 
> Right.  You need a dispatching layer behind this.  The beauty of  
> zope.event is that it puts very little requirement on people who  
> publish events.

The problem is that it relies on some specific dispatching layer, and 
that layer would *also* have to be defined to use this for something 
useful like the start/stop events that are motivating this discussion. 
That is, if you dispatch on "isinstance(event, tuple) and event and 
event[0] == 'some_name'", that's very different than "isinstance(event, 
SomeClass)" or "implements(event, ISomething)" or whatever.  Actual 
servers have to emit something more specific than just "an object".


-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
             | Write code, do good | http://topp.openplans.org/careers

From fumanchu at amor.org  Mon Jun 25 23:46:30 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 14:46:30 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <F58D24E1-1132-40D7-9111-B7260905A64A@zope.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DCA8@ex9.hostedexchange.local>

Jim Fulton wrote:
> On Jun 25, 2007, at 4:51 PM, Robert Brewer wrote:
> 
> > Benji York wrote:
> >> Robert Brewer wrote:
> >>> I'd like to continue talking about standardization on
> >> site-wide process
> >>> state and services.
> >>
> >> Perhaps zope.event would be a good fit.
> >> http://svn.zope.org/zope.event/trunk/src/zope/event/README.txt
> >> ?view=markup
> >
> > All 4 lines of it? ;)
> >
> >     subscribers = []
> >
> >     def notify(event):
> >         for subscriber in subscribers:
> >             subscriber(event)
> >
> > Seriously, though, this handles the notification but not the state
> > machine, which I think is critical to the effort. It also doesn't  
> > do any
> > error-handling for misbehaving subscribers, so not all 
> subscribers are
> > guaranteed to run if there's an unhandled error in an earlier
> > subscriber. It also sends all events to all subscribers, instead of
> > partitioning the subscribers.
> 
> Right.  You need a dispatching layer behind this.  The beauty of  
> zope.event is that it puts very little requirement on people who  
> publish events.
> 
> > And finally, I don't want "import
> > zope.event" in my framework.
> 
> Why?  I have no a-priori objection to importing something that has  
> cherrypy in its name.

Because CP has 0 dependencies outside the stdlib right now, and adding
one when you have none is a lot more work (for us and our users) than
adding one when you already have one or more. Making that jump for four
lines of code seems excessive.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From fumanchu at amor.org  Mon Jun 25 23:47:11 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 14:47:11 -0700
Subject: [Web-SIG] Web Site Process Bus (re-send)
In-Reply-To: <20070625184846.EDC683A409B@sparrow.telecommunity.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DCAA@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> At 10:57 AM 6/25/2007 -0700, Robert Brewer wrote:
> >This should be a collaborative effort, and I'm very open to
> >discussion at all levels of detail. Even if this flies at the highest
> >conceptual level, there are still several things I know of 
> >we would need to nail down:
> >
> >  * States. For example, do we need an EXITING state? If so, 
> >we should probably rename STOPPED to IDLE.
> >  * State transitions: do we need to lock or block which Bus 
> >methods can be called, depending on the current state?
> >  * Standard non-state channels, such as signals. Do we want 
> >a 'status' channel (for e.g. zdaemon)?
> >  * Should bus.log take other args, like 'severity'? Should 
> >it assume a stdlib logging API?
> >  * Should bus.log be replaced with separate stdout and 
> >stderr instead?
> >  * ...and just to introduce a bikeshed, is there a better 
> >name for the spec effort than "WSPB"?
> >
> >I have lots of other questions, but I'll let you all ask them.
> 
> I have some more basic questions...  like, how do you know you need 
> to subscribe in the first place?  Who does the subscribing?
> 
> My initial reaction is that I like the general idea, but the idea of 
> having a bus *object* seems wrong to me somehow.  Perhaps because the 
> only way to get it seems to be in an already-running application, 
> which seems to sort of defeat the purpose.  But maybe my purposes are 
> different than yours?
> 
> Perhaps if you could present some example scenarios to show how you 
> see this actually being used...?

Sure. Let's assume I want to have a single host+port serve apps from two
different WSPB-enabled frameworks; let's use hypothetical future
versions of Pylons and Zope.

If I'm primarily a Pylons user, I'm used to starting my websites with
"paster serve". In this case, paste.script would create a WSPBus object.
[It's up to the Paste developers whether to distribute their own
wspbus.py module, or to require one via setuptools.] When Paste parses
the paste.deploy file and composes the WSGI stack, it somehow hands a
Bus reference to Pylons. Exactly how is up for debate; Pylons might
provide a hook for it (say, in app_globals.py), or Pylons components
might take the bus as a constructor arg.

Paste's ServeCommand object would load the WSGI components and server
(as it does now), but then, instead of calling server(app), it would
call bus.subscribe('start', server, app). Pylons and Zope would also
have the opportunity to subscribe listeners to the 'start' channel (and
other channels), such as database connections, access log files, etc.
Then, instead of calling server(app) directly, Paste would call
bus.start(). It could then call bus.block(states=states.STOPPED) instead
of trapping KeyboardInterrupt itself. If block() traps a KBInt or
SyExit, it sends the 'stop' message to all subscribers, regardless of
whether they were created by Paste, Pylons, or Zope. They all shut down
synchronously, and then Paste could feel much safer about calling
bus.exit(). If KBInt or SysExit are trapped by a Pylons or Zope
component, they would be expected to call bus.stop or bus.exit (which
would unblock Paste).

If I'm primarily a Zope user instead, I might start my website with
zdaemon. This would work exactly like the above, but the Bus object
would be instantiated and started by the zdaemon package. If I'm using
Graham's new mod_wsgi with Apache, I might expect it to create and
control the Bus.

If I'm using paste.script but I want autoreload to use execv instead of
the default child-process style of autoreload, this is much easier with
the WSPBus. Instead of hacking paste.script, I unsubscribe Paste's
default autoreloader from the 'restart' channel and subscribe my own.

I'll stop there for now.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From pje at telecommunity.com  Mon Jun 25 23:52:13 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 25 Jun 2007 17:52:13 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <468036B8.5060105@colorstudy.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<20070625184846.EDC683A409B@sparrow.telecommunity.com>
	<46801756.4080709@colorstudy.com>
	<20070625201558.31A3C3A409B@sparrow.telecommunity.com>
	<468036B8.5060105@colorstudy.com>
Message-ID: <20070625215008.1BCC03A40A8@sparrow.telecommunity.com>

At 04:42 PM 6/25/2007 -0500, Ian Bicking wrote:
>Phillip J. Eby wrote:
>>At 02:28 PM 6/25/2007 -0500, Ian Bicking wrote:
>>>Potentially a Zope-style minimal event framework would 
>>>work.  Maybe something like:
>>>
>>>   send_signal(signal_name, signal_data)
>>>   subscribe(signal_name, listener)
>>That was what I was wondering, too, except I was thinking it would 
>>be sufficient to use entry points for subscription, but only invoke 
>>the entry points whose modules are in sys.modules.  In other words, 
>>never actually import a module in order to invoke a callback.  That 
>>way, subscription is a natural side effect of importing the modules 
>>that contain the listeners.  Something like:
>>def send_signal(group, name, *args, **kw):
>>     for ep in iter_entry_points(group, name):
>>         if ep.module_name in sys.modules:
>>             ep.load()(*args, **kw)
>
>I don't think that makes sense for this case.  The way I imagine using it is:
>
>class MyApplication(object):
>
>     def __init__(self, db):
>         self.db = db
>         subscribe('reload_resources', self.reload)
>     def reload(self, data=None):
>         self.db.close()
>         self.db.open()


I guess I wasn't clear.  The point is that the approach I'm 
suggesting allows event subscribers to be decoupled.  A library could 
have an "on_reload" function, for example, that references a 
library-specific event system.  Some frameworks, after all, are going 
to have their own event systems.

So a bridge from the bus to zope.event, for example, could just 
create objects and notify() them, and frameworks that are okay with 
depending on zope.event can use that.


>That is, I subscribe one particular thing when it is applicable, not 
>because the library is on the system.

That's why the code I suggested only invokes entry points that are 
*already imported* -- i.e., that someone is using. 


From pje at telecommunity.com  Tue Jun 26 00:02:18 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 25 Jun 2007 18:02:18 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DBE6@ex9.hostedexchang
	e.local>
References: <20070625210646.7A0F23A409B@sparrow.telecommunity.com>
	<435DF58A933BA74397B42CDEB8145A860D37DBE6@ex9.hostedexchange.local>
Message-ID: <20070625220010.916453A409B@sparrow.telecommunity.com>

At 02:17 PM 6/25/2007 -0700, Robert Brewer wrote:
>Phillip J. Eby wrote:
> > At 01:51 PM 6/25/2007 -0700, Robert Brewer wrote:
> > >Seriously, though, this handles the notification but not the state
> > >machine, which I think is critical to the effort. It also
> > doesn't do any
> > >error-handling for misbehaving subscribers, so not all
> > subscribers are
> > >guaranteed to run if there's an unhandled error in an earlier
> > >subscriber.
> >
> > Again, it would be really helpful if you could provide some scenarios
> > that show why these things are important.  It's not immediately
> > obvious to me that they are.
>
>I replied with some in another email, it's just being slow.
>
> > For example, if an error occurs, isn't that an indication that the
> > component is broken?  Masking the failure just makes it less likely
> > the component will get fixed in a timely manner.
>
>Yes, the component is broken. However, at runtime, breakage in a
>CherryPy component shouldn't keep a Quixote component from, say,
>correctly freeing its DB connections.

In theory that makes sense, but in practice if you're using 
priorities because there is a dependency sequence involved, then you 
now have a new problem, since a component you're relying on having 
started or stopped first, is now violating its invariants.

I'm not opposed to logging or catching errors, but I am opposed (in 
the absence of more specific evidence) to allowing callbacks to 
propagate unhandled exceptions in the spec, or encouraging event 
senders to make heroic efforts in the face of unhandled 
exceptions.  Trying to recover from brokenness is generally not very 
likely to result in *less* breaking, IMO.


> > Meanwhile, if you get a start call, you must be starting, right?  So
> > why worry about the state?  It'd be simpler to just use
> > "before/during/after" messages the way Twisted does.  Your "block"
> > example could be replaced by waiting for the "after" message of the
> > desired state, for example.
>
>That's a possible way to go. My intention was to support both 1)
>examination of the state by external components (for operations other
>than 'block'--progress meters spring to mind) and 2) restrict some state
>transitions if necessary; for example, make bus.start() do nothing (or
>block) unless the state is "STOPPED".

Progress meters can be handled by callbacks, too.  As for the 
restrictions, who benefits?  ISTM that components need to manage 
their own lifecycles anyway and should be idempotent with respect to 
repeated transitions.

But I'll look for the use cases email before commenting further.  :)


From ianb at colorstudy.com  Tue Jun 26 00:00:43 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 25 Jun 2007 17:00:43 -0500
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625215008.1BCC03A40A8@sparrow.telecommunity.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<20070625184846.EDC683A409B@sparrow.telecommunity.com>
	<46801756.4080709@colorstudy.com>
	<20070625201558.31A3C3A409B@sparrow.telecommunity.com>
	<468036B8.5060105@colorstudy.com>
	<20070625215008.1BCC03A40A8@sparrow.telecommunity.com>
Message-ID: <46803B0B.2070406@colorstudy.com>

Phillip J. Eby wrote:
> At 04:42 PM 6/25/2007 -0500, Ian Bicking wrote:
>> Phillip J. Eby wrote:
>>> At 02:28 PM 6/25/2007 -0500, Ian Bicking wrote:
>>>> Potentially a Zope-style minimal event framework would work.  Maybe 
>>>> something like:
>>>>
>>>>   send_signal(signal_name, signal_data)
>>>>   subscribe(signal_name, listener)
>>> That was what I was wondering, too, except I was thinking it would be 
>>> sufficient to use entry points for subscription, but only invoke the 
>>> entry points whose modules are in sys.modules.  In other words, never 
>>> actually import a module in order to invoke a callback.  That way, 
>>> subscription is a natural side effect of importing the modules that 
>>> contain the listeners.  Something like:
>>> def send_signal(group, name, *args, **kw):
>>>     for ep in iter_entry_points(group, name):
>>>         if ep.module_name in sys.modules:
>>>             ep.load()(*args, **kw)
>>
>> I don't think that makes sense for this case.  The way I imagine using 
>> it is:
>>
>> class MyApplication(object):
>>
>>     def __init__(self, db):
>>         self.db = db
>>         subscribe('reload_resources', self.reload)
>>     def reload(self, data=None):
>>         self.db.close()
>>         self.db.open()
> 
> 
> I guess I wasn't clear.  The point is that the approach I'm suggesting 
> allows event subscribers to be decoupled.  A library could have an 
> "on_reload" function, for example, that references a library-specific 
> event system.  Some frameworks, after all, are going to have their own 
> event systems.
> 
> So a bridge from the bus to zope.event, for example, could just create 
> objects and notify() them, and frameworks that are okay with depending 
> on zope.event can use that.

I think it's just using pkg_resources functionality so that you don't 
need a common thing to subscribe to.  Which I guess is exactly why 
Robert wants this idea of different buses instead of a single bus; 
though it's vague to me how you plug the buses and listeners and whatnot 
together without some common code or place to put objects (entry points 
being one possibility, a single module with these functions another).  I 
guess you could have an entry point that indicates objects that want to 
know about any buses that are created.  It seems like a lot of 
indirection, though.

>> That is, I subscribe one particular thing when it is applicable, not 
>> because the library is on the system.
> 
> That's why the code I suggested only invokes entry points that are 
> *already imported* -- i.e., that someone is using.

Just because the module is imported doesn't mean that every case in the 
module exists.  E.g., in my example only on the instantiation of 
MyApplication would I listen for the event.

This means that the subscription has to be done even if no one emits the 
signal.  Which is possible if you start listening for lots of different 
things because we can't agree on names and you want to support lots of 
systems.  But I don't think that's going to happen.


-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
             | Write code, do good | http://topp.openplans.org/careers

From pje at telecommunity.com  Tue Jun 26 00:08:44 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 25 Jun 2007 18:08:44 -0400
Subject: [Web-SIG] Web Site Process Bus (re-send)
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DCAA@ex9.hostedexchang
	e.local>
References: <20070625184846.EDC683A409B@sparrow.telecommunity.com>
	<435DF58A933BA74397B42CDEB8145A860D37DCAA@ex9.hostedexchange.local>
Message-ID: <20070625220636.B9D923A409B@sparrow.telecommunity.com>

At 02:47 PM 6/25/2007 -0700, Robert Brewer wrote:
>If I'm primarily a Pylons user, I'm used to starting my websites with
>"paster serve". In this case, paste.script would create a WSPBus object.
>[It's up to the Paste developers whether to distribute their own
>wspbus.py module, or to require one via setuptools.] When Paste parses
>the paste.deploy file and composes the WSGI stack, it somehow hands a
>Bus reference to Pylons. Exactly how is up for debate; Pylons might
>provide a hook for it (say, in app_globals.py), or Pylons components
>might take the bus as a constructor arg.
>
>Paste's ServeCommand object would load the WSGI components and server
>(as it does now), but then, instead of calling server(app), it would
>call bus.subscribe('start', server, app). Pylons and Zope would also
>have the opportunity to subscribe listeners to the 'start' channel (and
>other channels), such as database connections, access log files, etc.
>Then, instead of calling server(app) directly, Paste would call
>bus.start(). It could then call bus.block(states=states.STOPPED) instead
>of trapping KeyboardInterrupt itself. If block() traps a KBInt or
>SyExit, it sends the 'stop' message to all subscribers, regardless of
>whether they were created by Paste, Pylons, or Zope. They all shut down
>synchronously, and then Paste could feel much safer about calling
>bus.exit(). If KBInt or SysExit are trapped by a Pylons or Zope
>component, they would be expected to call bus.stop or bus.exit (which
>would unblock Paste).
>
>If I'm primarily a Zope user instead, I might start my website with
>zdaemon. This would work exactly like the above, but the Bus object
>would be instantiated and started by the zdaemon package. If I'm using
>Graham's new mod_wsgi with Apache, I might expect it to create and
>control the Bus.

I sort of understand the above scenarios -- except the part where the 
bus is actually doing anything useful.  What is it that you get that 
you can't do almost as easily some other way?

(The other piece that throws me is the idea of using block to *run* 
the main process.  Huh?  Where's the event loop then?)


>If I'm using paste.script but I want autoreload to use execv instead of
>the default child-process style of autoreload, this is much easier with
>the WSPBus. Instead of hacking paste.script, I unsubscribe Paste's
>default autoreloader from the 'restart' channel and subscribe my own.

This sounds like something useful, or it would if I had any use for 
code reloaders, which IMO are more appropriate for a development 
environment than a production one.  Nonetheless, I'm not sure it 
deserves an entire event bus specification just to be able to have 
pluggable autoreload.  :)


From pje at telecommunity.com  Tue Jun 26 00:10:40 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 25 Jun 2007 18:10:40 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DC80@ex9.hostedexchang
	e.local>
References: <20070625210646.7A0F23A409B@sparrow.telecommunity.com>
	<435DF58A933BA74397B42CDEB8145A860D37DC80@ex9.hostedexchange.local>
Message-ID: <20070625220834.88ED43A409B@sparrow.telecommunity.com>

At 02:41 PM 6/25/2007 -0700, Robert Brewer wrote:
>Phillip J. Eby wrote:
> > Meanwhile, if you get a start call, you must be starting, right?  So
> > why worry about the state?  It'd be simpler to just use
> > "before/during/after" messages the way Twisted does.  Your "block"
> > example could be replaced by waiting for the "after" message of the
> > desired state, for example.
>
>I just realized I haven't really explained what "start" and "stop" mean.
>I think you might expect it to mean "beginning of process" and "exit the
>process". But instead, I'm envisioning a FSM that has an "idle" state
>in-between process init/exit and "server" start/stop, so that, without
>restarting the process, you can stop and restart (un/bind the socket,
>etc) the server components. This should also facilitate a daemon parent
>process having a single site Bus and starting/stopping child processes
>that contain the WSGI app and server components, if that's the way you
>want to compose your site.

Now I'm really confused.  What is the idle state *for*?  This seems 
to imply that you need these states to exist for distinct components 
in a single process -- which would be a finer-grained sort of "bus" 
than has been discussed at this point, at least as far as I understand it.


From ianb at colorstudy.com  Tue Jun 26 00:10:14 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 25 Jun 2007 17:10:14 -0500
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625220010.916453A409B@sparrow.telecommunity.com>
References: <20070625210646.7A0F23A409B@sparrow.telecommunity.com>	<435DF58A933BA74397B42CDEB8145A860D37DBE6@ex9.hostedexchange.local>
	<20070625220010.916453A409B@sparrow.telecommunity.com>
Message-ID: <46803D46.4010403@colorstudy.com>

Phillip J. Eby wrote:
> At 02:17 PM 6/25/2007 -0700, Robert Brewer wrote:
>> Phillip J. Eby wrote:
>>> At 01:51 PM 6/25/2007 -0700, Robert Brewer wrote:
>>>> Seriously, though, this handles the notification but not the state
>>>> machine, which I think is critical to the effort. It also
>>> doesn't do any
>>>> error-handling for misbehaving subscribers, so not all
>>> subscribers are
>>>> guaranteed to run if there's an unhandled error in an earlier
>>>> subscriber.
>>> Again, it would be really helpful if you could provide some scenarios
>>> that show why these things are important.  It's not immediately
>>> obvious to me that they are.
>> I replied with some in another email, it's just being slow.
>>
>>> For example, if an error occurs, isn't that an indication that the
>>> component is broken?  Masking the failure just makes it less likely
>>> the component will get fixed in a timely manner.
>> Yes, the component is broken. However, at runtime, breakage in a
>> CherryPy component shouldn't keep a Quixote component from, say,
>> correctly freeing its DB connections.
> 
> In theory that makes sense, but in practice if you're using 
> priorities because there is a dependency sequence involved, then you 
> now have a new problem, since a component you're relying on having 
> started or stopped first, is now violating its invariants.

I'm anti-priorities.  It's hard to arrange all those priorities in some 
useful way, and I don't see the purpose.

> I'm not opposed to logging or catching errors, but I am opposed (in 
> the absence of more specific evidence) to allowing callbacks to 
> propagate unhandled exceptions in the spec, or encouraging event 
> senders to make heroic efforts in the face of unhandled 
> exceptions.  Trying to recover from brokenness is generally not very 
> likely to result in *less* breaking, IMO.

There's also the weird situation where the exception raised depends on 
what the last subscriber is, or priority, or what components are loaded, 
or something like that depending on the implementation.

I might be inclined to turn the exception into another event.  Maybe:

def publish(self, channel, *args, **kw):
     subscribers = self.subscribers.get(channel, [])
     for subscriber in subscribers:
         try:
             subscriber(channel, *args, **kw)
         except:
             if channel == 'error':
                 raise
             exc_info = sys.exc_info()
             self.publish('error', subscriber, exc_info, (channel, args, 
kw))

In testing you could add a listener for 'error' that would just raise 
the exception, so that errors are never hidden in that case.  But 
otherwise, when there's code from different sources (e.g., a server 
hosting Moin, a Pylons app, etc) I'd rather one of those not be able to 
mess up the other too easily.

-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
             | Write code, do good | http://topp.openplans.org/careers

From fumanchu at amor.org  Tue Jun 26 00:18:29 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 15:18:29 -0700
Subject: [Web-SIG] Web Site Process Bus (re-send)
In-Reply-To: <20070625220636.B9D923A409B@sparrow.telecommunity.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DD53@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> Sent: Monday, June 25, 2007 3:09 PM
> To: Robert Brewer; web-sig at python.org
> Subject: RE: [Web-SIG] Web Site Process Bus (re-send)
> 
> At 02:47 PM 6/25/2007 -0700, Robert Brewer wrote:
> >If I'm primarily a Pylons user, I'm used to starting my websites with
> >"paster serve". In this case, paste.script would create a 
> WSPBus object.
> >[It's up to the Paste developers whether to distribute their own
> >wspbus.py module, or to require one via setuptools.] When 
> Paste parses
> >the paste.deploy file and composes the WSGI stack, it somehow hands a
> >Bus reference to Pylons. Exactly how is up for debate; Pylons might
> >provide a hook for it (say, in app_globals.py), or Pylons components
> >might take the bus as a constructor arg.
> >
> >Paste's ServeCommand object would load the WSGI components and server
> >(as it does now), but then, instead of calling server(app), it would
> >call bus.subscribe('start', server, app). Pylons and Zope would also
> >have the opportunity to subscribe listeners to the 'start' 
> channel (and
> >other channels), such as database connections, access log files, etc.
> >Then, instead of calling server(app) directly, Paste would call
> >bus.start(). It could then call 
> bus.block(states=states.STOPPED) instead
> >of trapping KeyboardInterrupt itself. If block() traps a KBInt or
> >SyExit, it sends the 'stop' message to all subscribers, regardless of
> >whether they were created by Paste, Pylons, or Zope. They 
> all shut down
> >synchronously, and then Paste could feel much safer about calling
> >bus.exit(). If KBInt or SysExit are trapped by a Pylons or Zope
> >component, they would be expected to call bus.stop or bus.exit (which
> >would unblock Paste).
> >
> >If I'm primarily a Zope user instead, I might start my website with
> >zdaemon. This would work exactly like the above, but the Bus object
> >would be instantiated and started by the zdaemon package. If 
> I'm using
> >Graham's new mod_wsgi with Apache, I might expect it to create and
> >control the Bus.
> 
> I sort of understand the above scenarios -- except the part where the 
> bus is actually doing anything useful.  What is it that you get that 
> you can't do almost as easily some other way?
> 
> (The other piece that throws me is the idea of using block to *run* 
> the main process.  Huh?  Where's the event loop then?)

It differs by framework. CP runs the socket-listening event loop in a
separate thread and blocks the main thread. If you were running the HTTP
server in the main thread, you'd have to make it the last subscriber to
'start' and let that block for you (that design isn't in the example
code on my blog, but it could be done fairly easily).

> >If I'm using paste.script but I want autoreload to use execv 
> instead of
> >the default child-process style of autoreload, this is much 
> easier with
> >the WSPBus. Instead of hacking paste.script, I unsubscribe Paste's
> >default autoreloader from the 'restart' channel and subscribe my own.
> 
> This sounds like something useful, or it would if I had any use for 
> code reloaders, which IMO are more appropriate for a development 
> environment than a production one.  Nonetheless, I'm not sure it 
> deserves an entire event bus specification just to be able to have 
> pluggable autoreload.  :)

The point is that *any* site-wide behavior would be pluggable, of
course, from the site log to platform-specific needs (daemonization
etc.), to "what does SIGUSR1 do?" These all seem to me like they'd
benefit from decoupling in order to foster a development market for
them. The CherryPy trunk even swaps out the entire Bus in order to use
win32events on that platform [1].


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

[1] http://www.cherrypy.org/browser/trunk/cherrypy/restsrv/win32.py

From graham.dumpleton at gmail.com  Tue Jun 26 00:53:28 2007
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 26 Jun 2007 08:53:28 +1000
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <46801756.4080709@colorstudy.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<20070625184846.EDC683A409B@sparrow.telecommunity.com>
	<46801756.4080709@colorstudy.com>
Message-ID: <88e286470706251553m662f0b96pc7ff83c159ed4d93@mail.gmail.com>

On 26/06/07, Ian Bicking <ianb at colorstudy.com> wrote:
> For instance, doesn't atexit work fine for exiting?  (Not in mod_python,
> but I still don't know why it *shouldn't* work in mod_python.)

Exit functions registered by atexit don't work in mod_python for a few reasons.

First off is that if you call Py_Finalize() then Python only calls
exit functions on the main Python interpreter and does not do it on
sub interpreters. Since code running under mod_python tends to run in
sub interpreters, any atexit registered functions would never be
called.

The second problem is that calling Py_Finalize() was disabled in more
recent versions of mod_python as it was causing lots of problems with
strange Python exceptions and sometimes crashes occurring when
triggered.

In mod_wsgi I managed to work out what all the problems were and in
mod_wsgi it will correctly call atexit registered functions registered
in all sub interpreters as well as call Py_Finalize() to clean up
other stuff related to any interpreters which were created. The
changes in mod_wsgi even deal with changes made in Python 2.5 with how
threads are killed off on killing a sub interpreter. Prior to 2.5 this
was managed by an atexit registered function from threading module
itself, but in 2.5 the Python C code itself triggers the cleanup
actions for threads.

Going back and fixing all this in mod_python is right down at the end
of the reopened issue for Py_Finalize() causing problems in the first
place.

  https://issues.apache.org/jira/browse/MODPYTHON-109

All the code in mod_python for Python initialisation and shutdown,
plus interpreter management really needs to be completely rewritten as
it is messy and still has bugs in there related to simplified GIL API,
something which has again been fixed up in mod_wsgi.

Graham

From fumanchu at amor.org  Tue Jun 26 01:07:53 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Mon, 25 Jun 2007 16:07:53 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625220834.88ED43A409B@sparrow.telecommunity.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D37DE3D@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> At 02:41 PM 6/25/2007 -0700, Robert Brewer wrote:
> >Phillip J. Eby wrote:
> > > Meanwhile, if you get a start call, you must be starting, 
> right?  So
> > > why worry about the state?  It'd be simpler to just use
> > > "before/during/after" messages the way Twisted does.  Your "block"
> > > example could be replaced by waiting for the "after" 
> message of the
> > > desired state, for example.
> >
> >I just realized I haven't really explained what "start" and 
> "stop" mean.
> >I think you might expect it to mean "beginning of process" 
> and "exit the
> >process". But instead, I'm envisioning a FSM that has an "idle" state
> >in-between process init/exit and "server" start/stop, so 
> that, without
> >restarting the process, you can stop and restart (un/bind the socket,
> >etc) the server components. This should also facilitate a 
> daemon parent
> >process having a single site Bus and starting/stopping child 
> processes
> >that contain the WSGI app and server components, if that's 
> the way you
> >want to compose your site.
> 
> Now I'm really confused.  What is the idle state *for*?

One concrete use case could be a test suite that swaps out applications
between tests without exiting the entire process. But there are other
situations where it's useful. The startup script might need a chance to
run mandatory code in-between stop and exit, or a debugger might stop a
live server (stop accepting connections), run a debug session, fix the
problem, then start up again.

> This seems to imply that you need these states to exist for
> distinct components in a single process -- which would be a
> finer-grained sort of "bus" than has been discussed at this
> point, at least as far as I understand it.

That's not what I had in mind, but I wouldn't be opposed to it.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From chad at zetaweb.com  Tue Jun 26 02:32:38 2007
From: chad at zetaweb.com (Chad Whitacre)
Date: Mon, 25 Jun 2007 20:32:38 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DE3D@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D37DE3D@ex9.hostedexchange.local>
Message-ID: <46805EA6.7060006@zetaweb.com>

All,

>> Now I'm really confused.  What is the idle state *for*?
> 
> One concrete use case could be a test suite ...

Is this also where Joe's comment in Dallas fits, namely, the case 
where you want to hold open a socket while restarting a child 
process so as not to drop connections?



chad

From graham.dumpleton at gmail.com  Tue Jun 26 04:48:47 2007
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 26 Jun 2007 12:48:47 +1000
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
Message-ID: <88e286470706251948x11efa239w5e9cf64000a8f6d0@mail.gmail.com>

On 26/06/07, Robert Brewer <fumanchu at amor.org> wrote:
> I'd like to continue talking about standardization on site-wide process
> state and services.
>
> As described more fully on my blog [1], I'm proposing we create a new
> spec for a simple publish-subscribe Bus to manage site-wide state
> transitions (start, stop, etc.) and connect application components with
> site-wide services (daemonize, autoreload, site logging, etc).

I'm a bit too busy to fully absorb all that is being said on this
topic, but have a few comments or observations to make based on my
experiences. If they relate to anything, it is possibly just to expand
on some of the issues around why a bus might be a reasonable
requirement to have so hopefully it is of use in that respect.

First comment is about WSGI applications somehow themselves using
SIGTERM etc as triggers for things they want to do. For Apache at
least, allowing any part of a hosted Python application to register
its own signal handlers is a big no no. This is because Apache itself
uses a whole range of signals to manage such tasks as shutting down
sub processes or signaling worker and/or listener threads within a
process that its time to wakeup or shutdown. If a WSGI application
starts registering signal handlers it can as a result stop Apache from
even being able to process requests. In mod_wsgi I have had to
specifically take steps to prevent applications breaking things in
this way by replacing signal.signal() on creation of an interpreter.
Instead I log a warning that the signal registration has been ignored
and otherwise do nothing. This was simply the safest thing to do.

Thus I believe a clear statement should be made that UNIX signals are
off limits to WSGI applications or components. I have no problem with
the underlying server upon which the WSGI adapter sits using them for
its own purposes, however, knowledge of signals should stay in that
layer. Now it may be the case that a higher level application signal
may be generated as a result of a SIGTERM in a specific server
implementation, for example shutdown, but the name or type of signal
does have to be distinct, ie., "shutdown" and have no linkage to the
UNIX signal.

In Apache for example, SIGTERM isn't used in child processes to
indicate process shutdown, that is instead signalled  via what is
called the pipe of death. The trigger for process shutdown in this
case is actually the destruction of the child process memory pool.

I know that Robert's diagram shows the translation of UNIX signal at a
lower level to higher level concept, but then elsewhere there was
mention of SIGUSR signals. Apache even uses these in some
configurations so a hosted user application wouldn't even be able to
use them.

Anyway, just wanted to make it absolutely clear that I don't believe a
hosted WSGI application and associated framework has any business
taking direct interest in low level UNIX signals.

Next issue was SystemExit exception, raised explicitly or via
sys.exit(), and the ability of a WSGI application to itself shutdown
the server. Again, shutting down the server using SystemExit is
something I see as being off limits to a WSGI application. In mod_wsgi
I explicitly ignore SystemExit raised by a WSGI application when
handling a request, more or less treating it like any other unexpected
exception. This is necessary because within Apache one can't  tolerate
one application making a decision to shutdown a process because of the
impacts it may have on other application code, possibly non Python
code, running in the same process.

For a Python based web server, also believe that SystemExit in a
request handler should be ignored. Obviously though the underlying
server itself is free to use its for its own purpose. I actually don't
know what other WSGI adapters do about SystemExit, so would be
interested to learn what others do.

One thing I have sometimes thought about though is whether there is
any valid case where a WSGI application should itself be allowed to
request the application be shutdown. I think I'd probably have to say
no for the general case. But then you have the case of trying to
reload an application when the code changes. In this case it may be
reasonable because no in process module reloading scheme is ever going
to be good enough. The closest one can get doing it in process is to
destroy the whole sub interpreter. This though is only a practical
thing for something like mod_wsgi and even then doesn't always work
because of C extension modules, eg PyProtocols, that don't take kindly
to sub interpreters being destroyed and the module then being reloaded
into a new sub interpreter.

Thus, purely to make administration easier for the WSGI application to
request an application reload, perhaps triggered by user through a web
page, one might want to signal process shutdown. Even then though it
isn't that simple as that, as there has to be a compatible supervisor
system in place which will restart the process. In mod_wsgi if using
it in embedded mode whereby application runs in Apache child
processes, an application requesting a reload like this is not viable
as not possible to tell the Apache parent process to restart
everything.

Even when using daemon mode of mod_wsgi where application runs in
separate processes, only the simple case where there is one daemon
process for that application is it trivial to initiate a reload. When
there are multiple processes in the group, need to somehow signal the
other processes in the group to also reload.

So that is my brain dump. I guess I'll see where you guys take the
discussion. At the moment I am guarded as to whether some of the ideas
of how this bus could be used would in practice be transferable.
Certainly with Apache it may be quite challenging because of the
multiprocess nature of the server and the fact that child processes
don't really have any way of controlling the supervisor (parent Apache
process).

Graham

From jtate at rpath.com  Tue Jun 26 04:51:16 2007
From: jtate at rpath.com (Joseph Tate)
Date: Mon, 25 Jun 2007 22:51:16 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <46805EA6.7060006@zetaweb.com>
References: <435DF58A933BA74397B42CDEB8145A860D37DE3D@ex9.hostedexchange.local>
	<46805EA6.7060006@zetaweb.com>
Message-ID: <200706252251.16703.jtate@rpath.com>

On Monday 25 June 2007 20:32:38 Chad Whitacre wrote:
> All,
>
> >> Now I'm really confused.  What is the idle state *for*?
> >
> > One concrete use case could be a test suite ...
>
> Is this also where Joe's comment in Dallas fits, namely, the case
> where you want to hold open a socket while restarting a child
> process so as not to drop connections?

It's not so much a desire to not drop connections as to keep the socket file 
descriptor during a restart to prevent an asynchronous kernel call 
(socket.close()) from preventing your new process from reopening that socket.  
The frequent work around is to insert an arbitrary sleeps or loops on the 
socket.bind call, but I store the file descriptor as an environment variable 
and reconnect to the existing socket after the exec call.  This is a problem 
on Linux at least, and probably all POSIX systems.  Having the kernel queue 
up connections is an added bonus.

I do think this is a good example of what the IDLE state could be used for.

-- 
Joseph Tate
Software Engineer
rPath Inc.
http://www.rpath.com/rbuilder/
(919) 851-3984 x2106

From jtate at rpath.com  Tue Jun 26 06:06:34 2007
From: jtate at rpath.com (Joseph Tate)
Date: Tue, 26 Jun 2007 00:06:34 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <88e286470706251948x11efa239w5e9cf64000a8f6d0@mail.gmail.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<88e286470706251948x11efa239w5e9cf64000a8f6d0@mail.gmail.com>
Message-ID: <200706260006.34400.jtate@rpath.com>

On Monday 25 June 2007 22:48:47 Graham Dumpleton wrote:
> Thus I believe a clear statement should be made that UNIX signals are
> off limits to WSGI applications or components. I have no problem with
> the underlying server upon which the WSGI adapter sits using them for
> its own purposes, however, knowledge of signals should stay in that
> layer. Now it may be the case that a higher level application signal
> may be generated as a result of a SIGTERM in a specific server
> implementation, for example shutdown, but the name or type of signal
> does have to be distinct, ie., "shutdown" and have no linkage to the
> UNIX signal.

I think this is fine.  Signal handlers only get run in the main thread anyway, 
so ignoring them for sub-threads (like handlers, dispatchers, etc) is a 
natural division.

<snip lots of interesting things about SystemExit>
A web bus obviates the need to signal other processes in the process group.  
I'm not clear on how the web bus devolves in the multi-process state though.

My particular use case for keeping SystemExit around is I have an app that 
allows the user to upload a new SSL certificate.  Without a restart, or 
perhaps, given a WSPBus,  just a drop to IDLE state, the new SSL certificate 
would not be applied to new incoming connections.  The thread that puts the 
new SSL Certificate in place needs to be able to tell the entire server to 
reload.

A second use case has already been mentioned: reloading running code.  The 
same app as above runs the system software update mechanism.  When that 
update completes (which could contain a new core c library, new python, new 
modules, new application code, or even new configuration data) the server 
needs to reload.

I think that SystemExit should not be removed from the WSGI application layer, 
unless it's replaced with another mechanism that allows the working 
thread/process to signal that the entire server should be restarted/reloaded.

-- 
Joseph Tate
Software Engineer
rPath Inc.
http://www.rpath.com/rbuilder/
(919) 851-3984 x2106

From graham.dumpleton at gmail.com  Tue Jun 26 07:04:28 2007
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 26 Jun 2007 15:04:28 +1000
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <200706260006.34400.jtate@rpath.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<88e286470706251948x11efa239w5e9cf64000a8f6d0@mail.gmail.com>
	<200706260006.34400.jtate@rpath.com>
Message-ID: <88e286470706252204v298648d3id73a660362074aac@mail.gmail.com>

On 26/06/07, Joseph Tate <jtate at rpath.com> wrote:
> On Monday 25 June 2007 22:48:47 Graham Dumpleton wrote:
> > Thus I believe a clear statement should be made that UNIX signals are
> > off limits to WSGI applications or components. I have no problem with
> > the underlying server upon which the WSGI adapter sits using them for
> > its own purposes, however, knowledge of signals should stay in that
> > layer. Now it may be the case that a higher level application signal
> > may be generated as a result of a SIGTERM in a specific server
> > implementation, for example shutdown, but the name or type of signal
> > does have to be distinct, ie., "shutdown" and have no linkage to the
> > UNIX signal.
>
> I think this is fine.  Signal handlers only get run in the main thread anyway,
> so ignoring them for sub-threads (like handlers, dispatchers, etc) is a
> natural division.
>
> <snip lots of interesting things about SystemExit>
> A web bus obviates the need to signal other processes in the process group.

It depends.

> I'm not clear on how the web bus devolves in the multi-process state though.

I guess with mod_wsgi I have the worst case scenario as I have to deal
with not only multiple processes, but also the possible presence of
multiple sub interpreters within a process and many distinct WSGI
applications being run in parallel.

Consider a few use case examples. First one is where Apache is being
shutdown or restarted. To tell the child process to shutdown, the main
process will send as many characters onto the pipe of death as there
are child processes. Each child process reads one character when it
sees data is available on the pipe of death and then shuts down the
process.

As part of the child process shutdown it will trigger destruction of
the child memory pool. To this would be registered cleanup callbacks,
one of which is a callback into mod_wsgi. From that callback all the
sub interpreters which have been created will in turn have any threads
cleaned up and then exit funcs called.

Within this bus scheme, prior to any threads being cleaned up,
mod_wsgi would need to signal any subscribed parties in the sub
interpreter that the the process (server) is being shutdown.

Now having a bus only know about stuff local to the process in this
case is fine as the propagation of the event comes from outside and
ends up inside. The problem is cases where it may be valid that the
source of the signal is actually inside of the WSGI application.

The issue here is one of scope and how does one know how far the event
should propagate. In mod_wsgi you have the decision as to whether the
event should only be propagated within the same sub interpreter,
whether it makes sense to propagate it to other sub interpreters
within the same process, or finally whether other process also need to
be notified. When it hits the other processes, does it go to all sub
interpreters, or just the interpreter with the same name as the
originating application in the other process, ie., where another copy
of the same application would be running.

In practice I think the bus has to be restricted so as to be as simple
as possible and for external packages to be used if more sophisticated
signalling between multiprocess applications is required. But then in
saying that, making it simple and keeping it just to that sub
interpreter or process may actually prevent some of the things being
discussed being feasible, at least in the context of Apache.

The only way we may know is to start stepping through specific uses
cases one at a time, not even worrying about the mechanisms of how the
bus may actually work, and discuss what would the intention of each
be.

> My particular use case for keeping SystemExit around is I have an app that
> allows the user to upload a new SSL certificate.  Without a restart, or
> perhaps, given a WSPBus,  just a drop to IDLE state, the new SSL certificate
> would not be applied to new incoming connections.  The thread that puts the
> new SSL Certificate in place needs to be able to tell the entire server to
> reload.

In Apache changing the certificates would need a complete restart of
everything. Because the  child processes aren't privileged they would
not be able to trigger the main server to do so. This actually gets to
one of my reservations about some of the stuff being discussed. That
is, that the WSGI applications should even have any ability to control
the underlying web server. In a shared web hosting environment using
Apache, allowing such control is not practical as you don't want
arbitrary user doing things to the server. If you are running Apache
as a dedicated server for a single application that is a different
matter however. Thus some aspects of what can be done by via the bus
would  have to be controllable dependent on the environment in which
one is running.

At least with Apache, even initiating this sort of stuff from inside
of a WSGI application may not make a great deal of sense even then. It
would be far easier and preferable in Apache to use a suexec CGI
script to accept the upload of the SSL certificate and then trigger a
restart of Apache. So in the end the bus concept may be great for pure
Python system, but not so sure about a complicated mixed code system
like Apache, especially where there may be better ways of handling it
through other features of Apache.

Graham

From fumanchu at amor.org  Tue Jun 26 18:38:22 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Tue, 26 Jun 2007 09:38:22 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <88e286470706252204v298648d3id73a660362074aac@mail.gmail.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D40DEA2@ex9.hostedexchange.local>

Graham, thanks for this and the previous post. I've had these issues in
mind while designing the current bus, but you raised them far more
eloquently than I could have.

Graham Dumpleton wrote:
> The only way we may know is to start stepping through specific uses
> cases one at a time, not even worrying about the mechanisms of how the
> bus may actually work, and discuss what would the intention of each
> be.
>
> > My particular use case for keeping SystemExit around is 
> > I have an app that allows the user to upload a new SSL
> > certificate.  Without a restart, or perhaps, given a WSPBus,
> > just a drop to IDLE state, the new SSL certificate would
> > not be applied to new incoming connections.  The thread that
> > puts the new SSL Certificate in place needs to be able to
> > tell the entire server to reload.
> 
> In Apache changing the certificates would need a complete restart of
> everything. Because the child processes aren't privileged they would
> not be able to trigger the main server to do so. This actually gets to
> one of my reservations about some of the stuff being discussed. That
> is, that the WSGI applications should even have any ability to control
> the underlying web server. In a shared web hosting environment using
> Apache, allowing such control is not practical as you don't want
> arbitrary user doing things to the server. If you are running Apache
> as a dedicated server for a single application that is a different
> matter however. Thus some aspects of what can be done by via the bus
> would  have to be controllable dependent on the environment in which
> one is running.

Right. My expectation was that Apache interfaces (like mod_wsgi and
modpython_gateway) would supply custom Bus objects which deny certain
behaviors (like calling bus state-transition methods from a WSGI
application). I think there's room for Apache site admins to choose
whether applications are allowed to do dangerous things, much like how
AllowOverride works for .htaccess.

> At least with Apache, even initiating this sort of stuff from inside
> of a WSGI application may not make a great deal of sense even then. It
> would be far easier and preferable in Apache to use a suexec CGI
> script to accept the upload of the SSL certificate and then trigger a
> restart of Apache.

That's not contrary to the bus concept. If there's a preferred way of
doing things, then a function can be written to do that, supplied by the
site interface, and be subscribed to the appropriate channel.

> So in the end the bus concept may be great for pure
> Python system, but not so sure about a complicated mixed code system
> like Apache, especially where there may be better ways of handling it
> through other features of Apache.

Cannot those "other features" be comoponentized? The only thing the Bus
tries to do is make a common interface for such behaviors--if Apache has
native methods to achieve the desired behavior, then great! Wrapping
them in bus listeners (and subscribing a safe set of them by default)
allows deployers who aren't familiar with Apache to get their site up
and running faster. However, mod_wsgi can still use Apache directives
for attaching/detaching the listeners if it likes, providing a more
Apache-like look-and-feel.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From chrism at plope.com  Tue Jun 26 19:00:07 2007
From: chrism at plope.com (Chris McDonough)
Date: Tue, 26 Jun 2007 13:00:07 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <88e286470706252204v298648d3id73a660362074aac@mail.gmail.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<88e286470706251948x11efa239w5e9cf64000a8f6d0@mail.gmail.com>
	<200706260006.34400.jtate@rpath.com>
	<88e286470706252204v298648d3id73a660362074aac@mail.gmail.com>
Message-ID: <57AD9275-13F3-4661-A205-9F5A67487939@plope.com>

On Jun 26, 2007, at 1:04 AM, Graham Dumpleton wrote:
> In Apache changing the certificates would need a complete restart of
> everything. Because the  child processes aren't privileged they would
> not be able to trigger the main server to do so. This actually gets to
> one of my reservations about some of the stuff being discussed. That
> is, that the WSGI applications should even have any ability to control
> the underlying web server. In a shared web hosting environment using
> Apache, allowing such control is not practical as you don't want
> arbitrary user doing things to the server. If you are running Apache
> as a dedicated server for a single application that is a different
> matter however. Thus some aspects of what can be done by via the bus
> would  have to be controllable dependent on the environment in which
> one is running.
>
> At least with Apache, even initiating this sort of stuff from inside
> of a WSGI application may not make a great deal of sense even then. It
> would be far easier and preferable in Apache to use a suexec CGI
> script to accept the upload of the SSL certificate and then trigger a
> restart of Apache. So in the end the bus concept may be great for pure
> Python system, but not so sure about a complicated mixed code system
> like Apache, especially where there may be better ways of handling it
> through other features of Apache.

There are also non-webbish processes like postgres, mysql, etc. that  
need to be treated as "part of the application".

I handle this currently by running all of the processes related to a  
specific project under a process controller (which happens to be  
implemented in Python, but that's besides the point, see http:// 
www.plope.com/software/supervisor2/).  The process controller is  
responsible for execing the child processes upon its own startup.  It  
is also responsible for restarting children if they die, capturing  
their output (if any), and allowing sufficiently privileged users to  
start and stop each one independently.  The only promise a subprocess  
must make to be managed is that it must be possible to start the  
process "in the foreground" (not under its down daemon manager).

If a "process bus" is implemented I suspect it should be implemented  
at this kind of level.  "Actions" could be registered for a specific  
subprocess types to send some input to a pipe file descriptor, send a  
signal to the process, etc.  It would also be possible to create some  
sort of dependency map between processes in a configuration, that  
relate the actions of one process to another (restart process A if  
process B is restarted, send a signal S to process C if signal T is  
sent to process D, etc).

- C


From ianb at colorstudy.com  Tue Jun 26 19:31:42 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 26 Jun 2007 12:31:42 -0500
Subject: [Web-SIG] Paste releases
Message-ID: <46814D7E.50102@colorstudy.com>

I've just made releases for Paste (core/Script/Deploy).  Quick summary:

Paste core:

* The paste.httpserver has neat new threadpool stuff to improve the
reliability of your threaded server.  Kill threads, see wedged threads,
restart the process, and more.  This has been in progress for some time
(and has held up releases), but after having it in production for a
month or so (from trunk -- it's okay for *me* to deploy from trunk ;)
I'm feeling fairly comfortable about it.

* Save HTTP proxy information, so you can in theory write HTTP proxies
with paste.httpserver

* More methods in TestApp, for your RESTful testing pleasure, like
app.put() and app.delete().

* paste.util.template has a few more constructs.

* Some request (object/function) bugs fixed.

In Paste Deploy:

* If you've had problems using filter-with and setup-app together (or
with appconfig() or any other problem with filter-with), it should be
better.

Paste Script only has one boring little change for logging.

easy_install -U to upgrade.  See the website for slightly more complete
news:
   http://pythonpaste.org/news.html
   http://pythonpaste.org/script/news.html
   http://pythonpaste.org/deploy/news.html


-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org
             | Write code, do good | http://topp.openplans.org/careers


From fumanchu at amor.org  Tue Jun 26 20:39:31 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Tue, 26 Jun 2007 11:39:31 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <57AD9275-13F3-4661-A205-9F5A67487939@plope.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D40E1AB@ex9.hostedexchange.local>

Chris McDonough wrote:
> There are also non-webbish processes like postgres, mysql, etc. that  
> need to be treated as "part of the application".
> 
> I handle this currently by running all of the processes related to a  
> specific project under a process controller (which happens to be  
> implemented in Python, but that's besides the point, see http:// 
> www.plope.com/software/supervisor2/).  The process controller is  
> responsible for execing the child processes upon its own startup.
> It is also responsible for restarting children if they die,
> capturing their output (if any), and allowing sufficiently
> privileged users to start and stop each one independently.
> The only promise a subprocess must make to be managed is that
> it must be possible to start the process "in the foreground"
> (not under its down daemon manager).
> 
> If a "process bus" is implemented I suspect it should be implemented  
> at this kind of level.

Ah, but there's the rub: we all have different ideas about how to
*implement* IPC and control. Which is why the WSPBus I outlined says
nothing about the mechanisms of message transport, RPC/IPC, or process
or thread boundaries. Instead, it defines the messages themselves, a set
of states for a given site, and a singleton message broker and state
machine. That's it.

If we can get that one piece into place, I think it can be a focal point
for interop between 1) application components, 2) HTTP servers, and 3)
"process controllers" (great term, I think I'll use it from now on). We
can achieve that without specifying how any process controller is
implemented, I think. It's difficult to discuss and reason about,
because CherryPy, Apache, Django, Zope, etcetera etcetera all provide
all three. A common bus should make it easier to decouple, say,
CherryPy's app+server from its process controller, allowing more people
to try out supervisor2 more easily.

> "Actions" could be registered for a specific subprocess types
> to send some input to a pipe file descriptor, send a signal to
> the process, etc.

Yes, and a supervisor2-wspb package could provide a Bus which does that,
then hand a reference to it to each child's components. The "reference"
in this case would be a child-side proxy object, which knows how to send
signals back to the parent process. But the children don't need to know
those transport details--all they have to do is call bus.subscribe,
bus.publish('log'), etc.

> It would also be possible to create some sort of dependency
> map between processes in a configuration, that relate the
> actions of one process to another (restart process A if
> process B is restarted, send a signal S to process C if
> signal T is sent to process D, etc).

Dependencies are a layer that can be built on top of the basic bus.
Since it's the process controller that's calling bus.start, stop and
restart, there's nothing about the WSPBus that stops supervisor2 from
handling dependency graphs on its own. If process A has to *know* that
process B has been restarted, that's a problem (which could be addressed
via custom bus channels), but if only the process controller has to
know, then there's no need to add that to the bus spec, IMO.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From chrism at plope.com  Tue Jun 26 21:47:03 2007
From: chrism at plope.com (Chris McDonough)
Date: Tue, 26 Jun 2007 15:47:03 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D40E1AB@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D40E1AB@ex9.hostedexchange.local>
Message-ID: <760BBB92-AAF0-4B38-B70A-F11EB03A27B6@plope.com>


On Jun 26, 2007, at 2:39 PM, Robert Brewer wrote:

> Chris McDonough wrote:
>> There are also non-webbish processes like postgres, mysql, etc. that
>> need to be treated as "part of the application".
>>
>> I handle this currently by running all of the processes related to a
>> specific project under a process controller (which happens to be
>> implemented in Python, but that's besides the point, see http://
>> www.plope.com/software/supervisor2/).  The process controller is
>> responsible for execing the child processes upon its own startup.
>> It is also responsible for restarting children if they die,
>> capturing their output (if any), and allowing sufficiently
>> privileged users to start and stop each one independently.
>> The only promise a subprocess must make to be managed is that
>> it must be possible to start the process "in the foreground"
>> (not under its down daemon manager).
>>
>> If a "process bus" is implemented I suspect it should be implemented
>> at this kind of level.
>
> Ah, but there's the rub: we all have different ideas about how to
> *implement* IPC and control.

I'm confused by this in your earlier message, describing example  
scenarios:

"""
If I'm primarily a Zope user instead, I might start my website with
zdaemon. This would work exactly like the above, but the Bus object
would be instantiated and started by the zdaemon package. If I'm using
Graham's new mod_wsgi with Apache, I might expect it to create and
control the Bus.
"""

I'm confused because zdaemon is a generic process controller, it  
knows nothing in particular about the application running under it  
except that it's a UNIX process.  It could start postgres instead of  
Zope if you configured it to.  If zdaemon creates a Bus object,  
nothing will be able to send messages to the bus except zdaemon  
itself, and there can't be any useful listeners because it doesn't  
share the same process space as its child.

I think I'm mostly confused by the name "process bus" because it  
seems like the primary use case for something like this is where all  
of the applications share the same process space and are all written  
in Python.  Am I right?  If so, maybe a different name is in order?   
"Application Bus"?  Or even "WSGI Bus", if its presumed that all of  
the applications will be WSGI applications?

- C


From pywebsig at xhaus.com  Tue Jun 26 22:04:53 2007
From: pywebsig at xhaus.com (Alan Kennedy)
Date: Tue, 26 Jun 2007 21:04:53 +0100
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <88e286470706251948x11efa239w5e9cf64000a8f6d0@mail.gmail.com>
References: <435DF58A933BA74397B42CDEB8145A860D37D6FB@ex9.hostedexchange.local>
	<88e286470706251948x11efa239w5e9cf64000a8f6d0@mail.gmail.com>
Message-ID: <46817165.60500@xhaus.com>

[Graham Dumpleton]
 > First comment is about WSGI applications somehow themselves using
 > SIGTERM etc as triggers for things they want to do. For Apache at
 > least, allowing any part of a hosted Python application to register
 > its own signal handlers is a big no no. This is because Apache itself
 > uses a whole range of signals to manage such tasks as shutting down
 > sub processes or signaling worker and/or listener threads within a
 > process that its time to wakeup or shutdown. If a WSGI application
 > starts registering signal handlers it can as a result stop Apache from
 > even being able to process requests. In mod_wsgi I have had to
 > specifically take steps to prevent applications breaking things in
 > this way by replacing signal.signal() on creation of an interpreter.
 > Instead I log a warning that the signal registration has been ignored
 > and otherwise do nothing. This was simply the safest thing to do.
 >
 > Thus I believe a clear statement should be made that UNIX signals are
 > off limits to WSGI applications or components.

 From a jython POV, I agree with this statement; signals don't even 
exist on java/jython (although some JVMs have non-standard extensions 
for signals).

Thus, any "standard" involving signals would not be implementable on 
jython, and I guess ironpython too.

[Graham Dumpleton]
 > Anyway, just wanted to make it absolutely clear that I don't believe a
 > hosted WSGI application and associated framework has any business
 > taking direct interest in low level UNIX signals.

Agreed.

Regards,

Alan.

From fumanchu at amor.org  Tue Jun 26 23:07:44 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Tue, 26 Jun 2007 14:07:44 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <760BBB92-AAF0-4B38-B70A-F11EB03A27B6@plope.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D40E51E@ex9.hostedexchange.local>

Chris McDonough wrote:
> On Jun 26, 2007, at 2:39 PM, Robert Brewer wrote:
> 
> > Chris McDonough wrote:
> >> There are also non-webbish processes like postgres, mysql, 
> >> etc. that
> >> need to be treated as "part of the application".
> >>
> >> I handle this currently by running all of the processes 
> >> related to a
> >> specific project under a process controller (which happens to be
> >> implemented in Python, but that's besides the point, see http://
> >> www.plope.com/software/supervisor2/).  The process controller is
> >> responsible for execing the child processes upon its own startup.
> >> It is also responsible for restarting children if they die,
> >> capturing their output (if any), and allowing sufficiently
> >> privileged users to start and stop each one independently.
> >> The only promise a subprocess must make to be managed is that
> >> it must be possible to start the process "in the foreground"
> >> (not under its down daemon manager).
> >>
> >> If a "process bus" is implemented I suspect it should be 
> >> implemented at this kind of level.
> >
> > Ah, but there's the rub: we all have different ideas about how to
> > *implement* IPC and control.
> 
> I'm confused by this in your earlier message, describing example  
> scenarios:
> 
> """
> If I'm primarily a Zope user instead, I might start my website with
> zdaemon. This would work exactly like the above, but the Bus object
> would be instantiated and started by the zdaemon package. If I'm using
> Graham's new mod_wsgi with Apache, I might expect it to create and
> control the Bus.
> """
> 
> I think I'm mostly confused by the name "process bus" because it  
> seems like the primary use case for something like this is where all  
> of the applications share the same process space

I don't see why it should be limited by that. The primary use case is
anywhere site components and application components are interacting,
that could benefit from a shared understanding (and control) of the
state of the site. To me, that requires a common set of messages, but
the transport mechanism for those messages should be flexible so that
it's useful in both multithread and multiprocess architectures.

> ...and are all written in Python.  Am I right?

That's the initial target market, yes. But I think we can design the
messaging spec to be useful with non-Python application components.

> If so, maybe a different name is in order?   
> "Application Bus"?  Or even "WSGI Bus", if its presumed that all of  
> the applications will be WSGI applications?

Sure, "application bus" is fine, although it's just the other side of
the same coin: "applications" on one side, "site" on the other.

I wouldn't want this to be "WSGI Bus", simply because there's no benefit
for that relationship; the two specs should be useful independently of
each other. In particular, we should be able to design a site-messaging
bus which works for WSGI 1.0, 1.1, 2.0, and whatever might obsolete the
current WSGI in the future.

> I'm confused because zdaemon is a generic process controller, it  
> knows nothing in particular about the application running under it  
> except that it's a UNIX process. It could start postgres instead of  
> Zope if you configured it to.

Sorry, I was really thinking of zopectl when I wrote that. I'm not sure
how zdaemon itself would fit into this whole scheme--it's the process
which zdaemon invokes that should be directly involved. If that process
is a good provider of bus-aware services, you might not need zdaemon
anymore.

> If zdaemon creates a Bus object,
> nothing will be able to send messages to the bus except zdaemon  
> itself, and there can't be any useful listeners because it doesn't  
> share the same process space as its child.

If you use the example Bus implementation I posted, then yes. That's why
I'm pitching WSPBus as a spec, not an implementation. Multiprocess
controllers could implement the bus using any of various forms of IPC;
they just need to arrange for each application component to get a Bus
object that, behind the scenes, is specific to the chosen method of IPC.

So, yes, interprocess communication is more complicated than
intraprocess. But that's true whether you standardize on a bus spec or
not.


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From chrism at plope.com  Tue Jun 26 23:38:14 2007
From: chrism at plope.com (Chris McDonough)
Date: Tue, 26 Jun 2007 17:38:14 -0400
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D40E51E@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D40E51E@ex9.hostedexchange.local>
Message-ID: <E4B2879E-CE1A-42F0-BC07-F771BF018677@plope.com>

On Jun 26, 2007, at 5:07 PM, Robert Brewer wrote:
>> I think I'm mostly confused by the name "process bus" because it
>> seems like the primary use case for something like this is where all
>> of the applications share the same process space
>
> I don't see why it should be limited by that. The primary use case is
> anywhere site components and application components are interacting,
> that could benefit from a shared understanding (and control) of the
> state of the site. To me, that requires a common set of messages, but
> the transport mechanism for those messages should be flexible so that
> it's useful in both multithread and multiprocess architectures.

Thank you.  I see.  This is a little too abstract for me to get my  
brain around, but I'll continue listening and maybe I'll get  
religion. ;-)

- C




From fumanchu at amor.org  Wed Jun 27 02:15:39 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Tue, 26 Jun 2007 17:15:39 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <20070625220010.916453A409B@sparrow.telecommunity.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D40E901@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> At 02:17 PM 6/25/2007 -0700, Robert Brewer wrote:
> > Phillip J. Eby wrote:
> > > At 01:51 PM 6/25/2007 -0700, Robert Brewer wrote:
> > > For example, if an error occurs, isn't that an indication that the
> > > component is broken?  Masking the failure just makes it 
> > > less likely the component will get fixed in a timely manner.
> >
> > Yes, the component is broken. However, at runtime, breakage in a
> > CherryPy component shouldn't keep a Quixote component from, say,
> > correctly freeing its DB connections.
> 
> In theory that makes sense, but in practice if you're using 
> priorities because there is a dependency sequence involved, then you 
> now have a new problem, since a component you're relying on having 
> started or stopped first, is now violating its invariants.
> 
> I'm not opposed to logging or catching errors, but I am opposed (in 
> the absence of more specific evidence) to allowing callbacks to 
> propagate unhandled exceptions in the spec, or encouraging event 
> senders to make heroic efforts in the face of unhandled 
> exceptions.  Trying to recover from brokenness is generally not very 
> likely to result in *less* breaking, IMO.

I agree with that in the general case, and specifically for site
startup, which is prone to dependencies and priorities. The specific
case where I feel we need different behavior is when trying to shut down
a site, which rarely involves dependencies and priorities, but can often
lead to increasing damage if an early component errors and remaining
resource cleanup routines are not allowed to run. Maybe we should just
special-case the latter and let the rest fail fast.

> > > Meanwhile, if you get a start call, you must be starting, 
> > > right?  So why worry about the state?  It'd be simpler to
> > > just use "before/during/after" messages the way Twisted does.
> > > Your "block" example could be replaced by waiting for the "after" 
> > > message of the desired state, for example.
> >
> > That's a possible way to go. My intention was to support both
> > 1) examination of the state by external components (for
> > operations other than 'block'--progress meters spring to mind)
> > and 2) restrict some state transitions if necessary; for example,
> > make bus.start() do nothing (or block) unless the state is
"STOPPED".
> 
> Progress meters can be handled by callbacks, too.

With sufficent complexity, yes.

> As for the restrictions, who benefits?  ISTM that components need
> to manage their own lifecycles anyway and should be idempotent
> with respect to repeated transitions.

Good point. Requiring idempotent operations would allow fun sequences
like:

>>> bus.exit()
>>> bus.exit()
>>> help!
SyntaxError: invalid syntax
>>> os.unlink(errant_lock_file)
>>> bus.exit()
...process finally exits...

That would also allow someone to call bus.exit() in the middle of
another thread executing bus.start()...if each component manages its own
state, that would minimize shutdown errors like closing DB connections
that were never opened.

Hmm...


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From fumanchu at amor.org  Thu Jun 28 00:09:23 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Wed, 27 Jun 2007 15:09:23 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D37DBE6@ex9.hostedexchange.local>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D4830B3@ex9.hostedexchange.local>

Phillip J. Eby wrote:
> Meanwhile, if you get a start call, you must be starting, right?
> So why worry about the state?  It'd be simpler to just use 
> "before/during/after" messages the way Twisted does.  Your "block" 
> example could be replaced by waiting for the "after" message of the 
> desired state, for example.

and I replied:
> That's a possible way to go. My intention was to support both 1)
> examination of the state by external components (for operations other
> than 'block'--progress meters spring to mind) and 2) restrict 
> some state
> transitions if necessary; for example, make bus.start() do nothing (or
> block) unless the state is "STOPPED".

Would it be helpful to just re-use the terms that Twisted does (in
IReactorCore)? The two structures are very similar:

Twisted			WSPBus
-------			------
core.running		bus.state == states.STARTED
stop()			stop()
'shutdown' events		'stop' channel listeners
'startup' events		'start' channel listeners
run()				start()

The only big difference being that IReactorCore.run also starts the main
loop, but that's assumed to be a separate step for WSPBus. Note that
IReactorCore.stop raises an error if not core.running, too.

I'll also note in passing that Twisted "during" and "after" triggers log
on error but don't crash...


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From fumanchu at amor.org  Thu Jun 28 00:47:09 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Wed, 27 Jun 2007 15:47:09 -0700
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <760BBB92-AAF0-4B38-B70A-F11EB03A27B6@plope.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D483153@ex9.hostedexchange.local>

Chris McDonough wrote:
> I think I'm mostly confused by the name "process bus" because it  
> seems like the primary use case for something like this is where all  
> of the applications share the same process space and are all written  
> in Python.  Am I right?  If so, maybe a different name is in order?   
> "Application Bus"?  Or even "WSGI Bus", if its presumed that all of  
> the applications will be WSGI applications?

Thinking about this some more, you're right that "process" is not the
appropriate term. This is about coordination between application/server
components and site-wide services, and I was using "process" redundantly
to mean "site". How about Web Site Event Bus instead?


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org

From graham.dumpleton at gmail.com  Thu Jun 28 06:44:02 2007
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Thu, 28 Jun 2007 14:44:02 +1000
Subject: [Web-SIG] Web Site Process Bus
In-Reply-To: <435DF58A933BA74397B42CDEB8145A860D4830B3@ex9.hostedexchange.local>
References: <435DF58A933BA74397B42CDEB8145A860D37DBE6@ex9.hostedexchange.local>
	<435DF58A933BA74397B42CDEB8145A860D4830B3@ex9.hostedexchange.local>
Message-ID: <88e286470706272144v3279bcadn9c331078134d2952@mail.gmail.com>

On 28/06/07, Robert Brewer <fumanchu at amor.org> wrote:
> Phillip J. Eby wrote:
> > Meanwhile, if you get a start call, you must be starting, right?
> > So why worry about the state?  It'd be simpler to just use
> > "before/during/after" messages the way Twisted does.  Your "block"
> > example could be replaced by waiting for the "after" message of the
> > desired state, for example.
>
> and I replied:
> > That's a possible way to go. My intention was to support both 1)
> > examination of the state by external components (for operations other
> > than 'block'--progress meters spring to mind) and 2) restrict
> > some state
> > transitions if necessary; for example, make bus.start() do nothing (or
> > block) unless the state is "STOPPED".
>
> Would it be helpful to just re-use the terms that Twisted does (in
> IReactorCore)? The two structures are very similar:
>
> Twisted                 WSPBus
> -------                 ------
> core.running            bus.state == states.STARTED
> stop()                  stop()
> 'shutdown' events               'stop' channel listeners
> 'startup' events                'start' channel listeners
> run()                           start()

A question about about the idea of bus.start() like event to indicate startup.

Problem with this is that under mod_wsgi the actual web server child
process has possibly started long before a request may come in which
targets a specific WSGI application. This is because loading of a WSGI
application is effectively done by lazy loading, ie., code file only
gets loaded when URL for a request maps to it.

This is different to where a Python based web server is used as
generally one would in the program script load in the WSGI application
before you even start the web server, as you would need to get the
application entry point to be able to construct the fixed URL entry
point for the root. Pylons and Paste may be an exception to this as
not sure at what point it actually will load things.

How do you see being able to handle a startup like event in that case
for a WSGI application when they aren't effectively being preloaded?
How would you notify just that one application when it does finally
get loaded, or do you?

Graham

From fumanchu at amor.org  Thu Jun 28 19:13:21 2007
From: fumanchu at amor.org (Robert Brewer)
Date: Thu, 28 Jun 2007 10:13:21 -0700
Subject: [Web-SIG] Web Bus event graphs
In-Reply-To: <88e286470706272144v3279bcadn9c331078134d2952@mail.gmail.com>
Message-ID: <435DF58A933BA74397B42CDEB8145A860D502D47@ex9.hostedexchange.local>

Graham Dumpleton wrote:
> A question about about the idea of bus.start() like event to 
> indicate startup.
> 
> Problem with this is that under mod_wsgi the actual web server child
> process has possibly started long before a request may come in which
> targets a specific WSGI application. This is because loading of a WSGI
> application is effectively done by lazy loading, ie., code file only
> gets loaded when URL for a request maps to it.
> 
> This is different to where a Python based web server is used as
> generally one would in the program script load in the WSGI application
> before you even start the web server, as you would need to get the
> application entry point to be able to construct the fixed URL entry
> point for the root. Pylons and Paste may be an exception to this as
> not sure at what point it actually will load things.
> 
> How do you see being able to handle a startup like event in that case
> for a WSGI application when they aren't effectively being preloaded?
> How would you notify just that one application when it does finally
> get loaded, or do you?

In terms of the "site event bus" model, I would just say that lazy
applications join the start/stop cycle a bit later. They miss the first
"start" notification, so they'd either have to not subscribe to the
'start' channel at all, or would have to call their start listeners
manually on load/first request.

> ...the actual web server child process has possibly started
> long before a request may come in...

That reminds me, I wanted to also discuss another potential channel pair
for managing per-thread resources. CherryPy has an (on_start_thread,
on_stop_thread) pair for registering such callbacks.

Currently, CP invokes *_thread events by checking thread ID's on each
request. If the thread ID has been seen before (there's a set of "seen
thread IDs"), nothing happens; if it hasn't been seen, then
on_start_thread listeners are invoked. Since that chunk of code has to
work with various multithread schemes, the on_stop_thread listeners
aren't called until server shutdown (!).

That's pretty inefficient on its own, but when several WSGI components
in the stack all maintain their own map of seen threads, it becomes
unwieldy pretty quickly. If "the site" could notify such listeners, it
would be more accurate ("thread stop" events would fire when the thread
actually stops) and take less memory, since the site controller would be
the only code with a thread map (and probably already has one anyway).

This isn't limited to threads, by the way. When people talk about
"per-thread" resources, that can usually be safely commuted to
"per-logical-process", where "logical process" encompasses threads,
processes (since they have a main thread, at least), and even tasklets
(Arnar Birgisson is working on a Stackless WSGI server as we speak).


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org