[Web-SIG] Collating follow-up on the future of WSGI

[Cory Benfield]

All,

Thanks so much for your feedback to my original request for comments on the future of WSGI. You provided a ton of really useful feedback: when printed out on my printer it ended up at about 50 pages of information that was really engaging reading. I also want to thank you all for keeping the tone of the discussion so positive. On an topic like this one it can get tricky and emotionally charged very easily, and you did a great job of avoiding that problem.

I spent a few hours this morning going over your feedback and trying to extract some common threads. Altogether I believe that most participants were mostly in agreement over the direction we should take, with a few outliers in each case. I’d like to summarise what I believe were the big take-aways from that discussion to confirm that I’ve understood everyone.

I’d also like the members of this SIG to take this opportunity to discuss these proposals more concretely. Rather than expressing our sentiments about WSGI and its future more generally, I want people to critique and offer opinions on *these specific proposals*. The goal here is to get an understanding of whether these are worth doing, how we need to prioritise the work, and whether there’s anything we’re missing.

The below is formatted in restructured text for clarity, because it’s quite a lot of information.


Concrete Proposals
==================

WSGI
----

Overall, there was strong and fairly unanimous sentiment that WSGI itself should more or less be left alone. Minor adjustments would be valuable, and we should pursue them, but the contributors do not believe that making a substantial revision to WSGI would be adviseable.

I therefore propose we revise the WSGI specification to WSGI 1.1 and then consider it "final". The following points were raised for revising WSGI.

Asynchronous WSGI
~~~~~~~~~~~~~~~~~

This was generally regarded as too substantial a change to shoehorn into WSGI.

Benoit proposed that we could achieve this change by adding a correlator to WSGI. This would allow servers to associate a given call to ``write()`` with a specific request/response (most useful in HTTP/2 where this could correspond to a stream ID). This would then allow WSGI to transition to a purely callback-based model that could in principle cohabitate with an async protocol in Python.

This proposal is worth highlighting not becuase I think we should pursue it with our revision of WSGI, but because it's worth considering for any future specification we come up with. Note also that Graham pointed out that this would require some careful rethinking of reads from ``wsgi.input``.

Server Push
~~~~~~~~~~~

We can support HTTP/2 server push using Link headers. This could optionally be supplemented by defining a WSGI extension that provides a callable for doing server push, which would be a mild improvement over the Link header approach. A simple proposal for how to do this can easily be implemented without revising the WSGI specification, and may not even need to be enshrined in a PEP. However, if we revise the WSGI PEP we may want to provide a small section that indicates how to add these headers.

Socket Escape Hatch
~~~~~~~~~~~~~~~~~~~

Aside from Benoit, server operators were unanimously dismissive of the idea of a socket 'escape hatch'. In general it seems like servers would not be capable of achieving this. I think, therefore, this idea is unworkable.

Chunked Transfer Encoding
~~~~~~~~~~~~~~~~~~~~~~~~~

It would be nice to formalise chunked transfer encoding in WSGI. Currently there is no way to signal to applications that chunked transfer encoding is in use by the client, or for the application to request it from the server.

This seemed to be a low priority work item, but if we can make this enhancement easily then it's worth considering.

Bytes and Unicode
~~~~~~~~~~~~~~~~~

Several contributors expressed dissatisfaction for PEP-3333's approach to headers (namely, Latin-1-encoded Unicode strings), and expressed a preference for using bytestrings. If we attempt this change, we have a backward-compatibility concern, so we may have to live with this decision.

Regardless, this should be taken as a warning sign for any other specification we attempt: more on that later.

REQUEST_URI environ variable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Multiple contributors expressed an interest in bringing this environment variable into WSGI directly, making it a required part of the environ dictionary. An alternative name for this was RAW_URI.

Header Joining and Name Normalization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Armin pointed out that PEP-3333 does not mention what should happen to normalize header names and to join header fields that appear multiple times in a header block. CGI does not appear to proscribe a behaviour here either.

A revision of PEP-3333 should cover how header names get normalised (where the answer is basically "like CGI"), and how servers should join headers together (and logically, how applications can split them). Mostly this is simply codifying existing practice.

Content Lengths
~~~~~~~~~~~~~~~

We should clarify in the new specification that an application that reads beyond the logical length of the request as given by CONTENT_LENGTH will have its reads return immediately with the empty string. Servers are required to police that logic. This is codifying existing practice, and would also make CONTENT_LENGTH purely advisory.

File Wrapper
~~~~~~~~~~~~

Graham identified two specific problems with ``wsgi.file_wrapper``: specifically, the bad example (which doesn't close the file descriptor properly), and the larger problem about the overly general specification of the ``file_wrapper`` as he's blogged about in the past.

In general, we may want to identify ``file_wrapper`` as a point of contention, and consider more fundamental revisions to this under-used part of the WSGI specification. We may also want to consider simply dropping it entirely, or pulling it out to an extension specification rather than leaving it in WSGI 1.1.

Websockets and HTTP/2
---------------------

Here, the consensus seemed to be that we should investigate whether we can come up with WSGI-equivalent APIs for Websockets and HTTP/2. The goal here would be to allow servers that are interested in offering richer APIs than WSGI can provide would be able to do so.

This is particularly valuable for HTTP/2, which can function totally fine when used with WSGI but which can have much richer functionality made available to the application via a different API if that application requires it.

As a SIG we should begin to consider what these APIs should look like. We should also bear in mind the more advanced asynchronous interfaces (see below), and ensure that we can easily proxy these APIs to the more advanced ones, as we can with WSGI to ASGI.

Meta API
~~~~~~~~

Graham suggested, along with these more advanced APIs, that we may want a meta-API or API discovery API that would allow servers to target a single entry point which then provides access to the callables for WSGI, the Websockets API, and the HTTP/2 API. We should bear this idea in mind, but I don't believe we should worry too much about whether we actually build it or not at this time: it can definitely follow the prototyping work on the HTTP/2 specific APIs.

Asynchronous Python
-------------------

There was an overall positive reaction to introducing *something* that fills the WSGI role for applications that use asynchronous Python frameworks.

Exactly how this would look is unclear at this stage, and this should be thought of as a long-term goal for this SIG. Amber Brown is clearly working on an idea, and we also have Andrew's work on ASGI, either of which could be used as a basis for a future PEP.

There was, however, a clear consensus that this should be a clean break from WSGI: while a server implementing this specification should be able to call into WSGI applications with a small shim, there is no requirement that an *application* using this specification should be callable from a WSGI server.

We should absolutely bear this in mind, and we should encourage specification work to be done here when things are ready, but this is definitely not going to happen as quickly as the other ideas above.



Please let me know what you think of this set of proposals. If everything is in good shape, I’ll begin working on some of these items, hopefully with the help of some of the other members of this SIG!

Cory
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/web-sig/attachments/20160119/d302c95e/attachment.sig>