[Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Cory Benfield]

> On 5 Jan 2016, at 00:12, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> 
> 
>> On 4 Jan 2016, at 11:27 PM, Cory Benfield <cory at lukasa.co.uk <mailto:cory at lukasa.co.uk>> wrote:
>> 
>> All,
>> 
>> **TL;DR: What do you believe WSGI 2.0 should and should not do? Should we do it at all?**
>> 
>> It’s a new year, and that means it’s time for another attempt to get WSGI 2.0 off the ground. Many of you may remember that we attempted to do this last year with Rob Collins leading the charge, but unfortunately personal commitments made it impossible for Rob to keep pushing that attempt forward.
> 
> Although you call this round 2, it isn’t really. Robert’s effort was not the first time someone has pushed a WSGI 2.0 variant. So this is more like being about round 5 or 6.
> 
> In part because of those repeated attempts by people to propose something and label it as WSGI 2.0, I am very cool on reusing the WSGI 2.0 moniker. You will find little or no mention of ‘WSGI 2.0’ as a label in:
> 
>     https://github.com/python-web-sig/wsgi-ng <https://github.com/python-web-sig/wsgi-ng>
> 
> That is probably somewhat due to my grumbling about the use of ‘WSGI 2.0’ back then.
> 
> Time has moved on and so the bad feelings and memories associated with the ‘WSGI 2.0’ label due to early failed efforts have faded, but I would still suggest avoiding the label ‘WSGI 2.0’ if at all possible.

Thanks for that feedback. Consider WSGI 2.0 a catch-all name for the purposes of this specific discussion (the “what do we want WSGI to be going forward” one). As you’ve suggested here, it’s entirely possible that the result of this discussion will be several PEPs/APIs, or none at all, and it’s entirely possible that none of them would be called WSGI 2.0.

> My general feeling is that if any proposed changes to the existing WSGI (PEP 3333) specification cannot be technically implemented on all existing WSGI server/adapter implementations that any new specification should not still be called WSGI.
> 
> In other words, even if many of these implementations may not be used much any more, it must be able to work, without needing to mark things as optional, on CGI, FASTCGI, SCGI, mod_wsgi, gunicorn, uWSGI, Waitress, etc etc.
> 
> This is purely to avoid the confusion whereby implementations cannot or choose not to implement any new specification. The last thing any WSGI server author wants is having to deal with a constant stream of questions and bug reports about not supporting an updated specification where technically it was never going to be possible. We have some obligation not to inflict this on what are, in nearly all cases, volunteers in the Open Source world who work on these things in their spare time and who are not doing it as part of their paid employment.

Can I clarify this requirement a bit? Are you wanting to say that any future version of WSGI must be entirely compatible with PEP 3333: that is, may not introduce optional features or change existing behaviour, only clarify? Please don’t mistake this for me challenging the idea: I’m wanting to get a good understanding of what you’re suggesting with this, not agreeing or disagreeing at this stage.

> For example, mod_wsgi already supports HTTP/2 by virtue of the fact that the mod_h2 module in Apache exists. The existing internal APIs of Apache and how mod_wsgi uses those means that HTTP/2 bridges into the WSGI world with no code changes to mod_wsgi.

Agreed. If all we want is to keep the request/response cycle intact, then WSGI supports H2 already. One possibility that has already been suggested here would be to define a HTTP/2 extension to WSGI, advertised in the environ dict, that allows the application to signal pushes to the server. This would be a fairly simple extension to write and implement.

> They are therefore two different APIs and so why WebSocket should be dealt with in a separate specification and not carry the WSGI label at all. A specific WSGI server could still support the new WebSocket API, but purely because it decides to support both in the same process. Not because the WebSocket API makes use of the WSGI specification.

That’s reasonable: I’d be happy to have websocket support either be a WSGI extension or, as you suggest here, a wholly new API. One difficulty with creating a new API from whole cloth is encouraging server authors to support it, but it’s certainly possible to do. I’d like to hear back from the uWSGI, gunicorn, and Twisted folks in addition to yourself about whether they’d be interested in implementing such a non-WSGI API.

>> - Graceful incremental adoption path - no upgrade-all-components requirement baked into the design.
> 
> It is hard to see what you expectations are here.
> 
> Prior attempts to force ASYNC into WSGI, and in some respects WebSockets through forcing raw fd access have not been practical. WSGI simply is not a good vehicle for it. Long term it is going to be much better to have new APIs for new WebSocket and HTTP/2 support.

This is an interesting point and I’d like to try to get you to elaborate if possible.

It’s my genuinely held belief that WSGI in its current form is likely going to be largely abandoned unless it becomes possible to combine it with a Python event loop of some kind. This applies even to plain HTTP/1.1 requests, which are still well-served by allowing the application process to take advantage of concurrent programming techniques without blocking the server waiting for the response.

It may be that you fundamentally disagree with me here: is that the case? The other interpretation I can see of this comment is that you’d rather we did not call a new, async-friendly HTTP/X API WSGI, but instead gave it some other name (ASGI is taken, so…fast-WSGI? ;) ). That is also totally reasonable: as I said above, I am not concerned about whether we call the work done by this SIG WSGI or not.

> The only even partly graceful path is perhaps first ignoring WebSockets and HTTP/2 and coming up with a more rich higher level abstraction for the complete Python web application entry point itself. So the idea above of a higher level object which defines hooks for startup/shutdown, passing configuration and also perhaps the querying of what protocols are supported by the application and even optionally what specific URL endpoints those protocols are active on. You could even have a application say where static file assets live so the server could host them itself via any more optimal methods than the application itself could use.

I’d like to draw application and server authors to this point you’ve raised. Do people believe, in general, that there is some advantage in a meta-gateway interface that allows registration of more tightly-scoped gateway interfaces, such as WSGI and a websocket specific interface? Or would you resist implementing such a thing? It would be a waste of time to build such a thing if no-one wants it, but if there’s interest in implementing with and playing with it, I think it’s a reasonable thing to have.

> 1. An optionally updated WSGI specification labelled as WSGI 1.1. This has got nothing really to do with the initiative to have a way to handle WebSockets and HTTP/2. It would simply to be integrate changes which were raised the last time the WSGI specification was updated, but which were passed over because a PEP was in the end rushed through just to deal with Python 3, ignoring other concerns. There are only a few changes which this would cover.
> 
> As I note above, this is optional. But if we are going to close out WSGI and not develop it further, would be nice to fix up some of the last problems with it.

On a personal level, if the updated WSGI specification did nothing more than codify current WSGI practice that would be fine, but otherwise if we were intending to consider WSGI basically “done” I don’t really see the advantage in revving the specification. If the view is that WSGI as-is will have one more change and then we’ll start looking to other approaches (be those other standardised gateway interfaces or less-standard all-inclusive frameworks that bundle their own servers), it’ll be hard to sell anyone on taking up the newly changed WSGI instead of investing time moving to the new world.

> 2. Develop a higher level abstraction for what is a Python web application. Thus hooks for startup/shutdown, passing configuration from the server, or querying back configuration from the application pertaining to supported protocols, along with what sub URLs protocols are supported on, and where static file assets may be that application may want the server to handle if that would be more performant.
> 
> I believe that such a new high level abstraction will provide a better framework to hang things off when we introduce new protocols.

I can get behind this idea, though I view it as strictly optional and if no-one wants to implement it I think we should save ourselves the work!

> 3. Separate WebSocket API.
> 
> Basically ignore existing WSGI specification completely. Come up with the best API one can for WebSocket interaction at the server level. This should not just be exposing a socket, but be a higher level abstraction involving passing of actual WebSocket messages.
> 
> By using higher level abstraction it allows a server to implement the details using whatever mechanisms best fit that server implementation.
> 
> 4. Separate HTTP/2 API.
> 
> Again, ignore existing WSGI specification complete. Come up with the best API one can for dealing with HTTP/2.
> 
> For (3) and (4) lets do these as being our holy grail. Rather than compromise by trying to work with WSGI, lets first come up with what would be our ideal. Then lets see how that can fit within existing servers, possibly integrated via the richer application abstraction of (2).

This seems like a reasonable approach: if the rest of the community is interested in it, I’d be happy to go along with it.

> It isn’t clear what you expect this PEP to include, but trying to push for a PEP so quickly is unrealistic. There is likely going to need to be a fair bit of discussion and with the fact that people have real jobs, or other obligations, history has shown that rushing to a PEP just disenfranchises people and they will not contribute due to the inability to do so in too short a time frame.

I apologise if I was unclear, but I am not pushing for a PEP anytime before the end of calendar year 2016. What I’m pushing for right now is to get people talking with the idea that we will keep taking steps forward. Sometime later this month or early next month I intend to stand up a porto-PEP (or more) that do nothing more than enumerate the goals of the work. That’ll give us a document to refer to going forward, but still give us time to work on concrete implementation and real problems.

I am happy for this to take as long as it takes, but I’m also going to put light pressure on to keep moving forward, lest we get bogged down in discussion forever and produce nothing.

Thanks so much for your feedback Graham!

Cory

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20160105/870ecc2a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/web-sig/attachments/20160105/870ecc2a/attachment-0001.sig>