[Async-sig] SansIO (Was: PEP: asynchronous generators)

[Cory Benfield]

> On 11 Aug 2016, at 17:56, Antoine Pitrou <antoine at python.org> wrote:
> 
> 1) implementing a protocol usually goes beyond parsing (which, it's
> true, can easily be done "sans IO”).

Yes, no question. And in fact, the hyper-h2 docs are very clear on this point: http://python-hyper.org/projects/h2/en/stable/basic-usage.html <http://python-hyper.org/projects/h2/en/stable/basic-usage.html>

You cannot just drop in hyper-h2 and expect anything sensible to happen. It needs to be hooked up to I/O, and the user needs to make some decisions themselves.

However, I’m totally happy to stand by my original point, which was: regardless of whether or not writing a protocol parser and state machine is “easy” to do without I/O, the Python community has not done that for the last twenty years. The fact that the current list of sans-IO implementations is three entries long, two of which are less than a year old, is a good indication of that point.

I don’t believe anyone is saying that sans-IO protocol implementations will remove *all* work from writing full-fledged implementations. Such a dream is impossible. But ideally they will remove a large chunk of the work.

> 2) many non-trivial protocols are stateful, at least at the level of a
> single connection; the statefulness may require doing I/O spontaneously
> (example: sending a keepalive packet). You can partly solve this by
> having a lower layer implementing the stateless parts ("sans IO") and an
> upper layer implementing the rest above it, but depending on the
> protocol it may be impossible to offer an *entire* implementation that
> doesn't depend on at least some notion of I/O.

So the spontaneous I/O question is an interesting one, not because it involves doing I/O so much as because your example fundamentally involves access to a *clock*. I haven’t had to deal with this yet, but I’ve been thinking about it a bit. My current conclusion is that a clock is basically an I/O tool: it’s a thing that needs to be controlled from the outside implementation. This is largely because when we want to use clocks what we really want to do is use *timers*, and timers are a flow control tool.

This means that they fit into the category of thing I alluded to briefly in my talk: they’re a thing that the sans-IO implementation can provide help and guidance with, but not ultimately do itself. Another example of this has been flow control management in HTTP/2: while the sans-IO implementation can do a lot of the work, fundamentally it still needs you to tell it “I just dealt with 100kB of data, please free up that much space in the window”.

There is no getting around this for almost any protocol, but that’s ok. Once again, the goal is not to subsume *everything* about a protocol: as you point out, that’s simply not possible. Instead, the goal is to subsume as much as possible.

> 3) the Protocol abstraction in asyncio (massively inspired from Twisted,
> of course) is a pragmatic way to minimize the I/O coupling of protocol
> implementations (and one of the reasons why I pushed for it during the
> PEP discussion): it still has some I/O-related elements to it (a couple
> callbacks on Protocol, and a couple methods on Transport), but in a way
> that makes ignoring them much easier than when using "streams", sockets
> or similar concepts.

I agree: this is why I used Twisted Protocols in my discussion with Nick in python-ideas. However, they do only *minimize* it. Most asyncio/Twisted Protocols quite happily issue writes to their transports willy-nilly, and also a great many of them create Futures (which makes sense: the abstraction into the coroutine world has to happen somewhere!). Once you create a Future, you are no longer “sans-IO”: a Future is an event loop construct.

(On a side note, this is why Twisted has a slight hypothetical edge in the “sans-IO” race: an asyncio.Future is an event-loop construct, but a twisted.internet.defer.Deferred is not. Deferreds work perfectly without an event loop, but a Future always requires one.)

The biggest problem, though, is that an asyncio Protocol is written like this:

class MyProtocol(asyncio.Protocol):
    pass

This provides substantial programmer baggage. Because asyncio has a blessed I/O model, it is very, very difficult for most programmers to think about writing a Protocol that isn’t going to be used that way. Even though, as I demonstrated with Twisted Protocols in python-ideas, they absolutely do not require an event loop if written carefully. This is part of why divorcing your protocol library from asyncio *entirely* (don’t even import it!) is helpful: it forces a clean, clear line in the sand that says “I do not care how you do I/O”. Twisted has been fighting this battle for years, and asyncio isn’t going to have a better time of it.

Cory


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160812/d6239d43/attachment-0001.html>