[Async-sig] APIs for backpressure (was: Re: Some thoughts on asynchronous API design in a post-async/await world)

[Cory Benfield]

> On 7 Nov 2016, at 19:54, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> I still haven't had time to read the whole thing yet (there's quite a lot to unpack here!) but I think that <https://github.com/twisted/tubes> might be of interest in examining ways to deal with backpressure that are more declarative; flows are set up ahead of time and then manipulated explicitly as flows, rather than relying on the imperative structure of pseudo-blocking in coroutines.

I’d like to fork the discussion off to talk about backpressure for a while. I think the issue of propagating backpressure is one that is really well suited to this list, because even *knowing* about backpressure is the marker of an engineer who has designed complex asynchronous systems. I suspect this list has a higher proportion of people capable of talking sensibly about backpressure propagation than most.

For my part, I actually think the backpressure discussion in Nathaniel’s post was the most interesting *mechanical* part of the post. Nathaniel has correctly identified that any system that does buffered writes/reads (e.g. asyncio and Twisted) needs powerful systems for propagating backpressure in a sensible way. If we want people to really develop resilient asynchronous systems in Python it must be possible for those developers to be able to sensibly propagate backpressure through the system, and ideally to fall into a pit of success whereby the things developers naturally do propagate backpressure as a matter of course.

Unfortunately I’d like to suggest that neither Twisted nor asyncio are in possession of really great APIs for exerting and managing backpressure. Twisted’s IPushProducer/IConsumer interface is present and moderately effective, but it has a few limits. The most frustrating limitation of this design is that it does not allow for easy construction of a pipeline: an object can implement IConsumer/IPushProducer in only one “direction”: that is, if you have a chain of objects A <-> B <-> C, B can propagate backpressure only to C or to A. That’s problematic for protocols like HTTP/2 which require balancing backpressure in multiple directions at once. An additional problem is that Twisted’s APIs here are one-to-one: that is, each B can only have one A and one C associated with it. That makes it very hard to do a fan-in/fan-out design with IPushProducer or IConsumer.

Both of these problems can be worked around of course: the creation of proxy objects or implicit interfaces between multiple objects can allow this to work, and that’s effectively what Twisted’s HTTP/2 layer does. But these are complex APIs, and they are definitely expert-oriented. On top of this, the unfriendliness of that interface means that developers are likely to consider it an optional part of their protocol implementation and will thus fail to exert backpressure at all.

(All of the problems of Twisted’s interface here apply to asyncio, by the by, I just happen to be able to talk in more detail about Twisted’s.)

Tubes is unquestionably a better API for this, but suffers from a lack of accessibility. It would definitely be interesting to see if tubes can be easily replicated on top of asyncio (and indeed curio), because at least for me something like tubes is what I’ve wanted for a long time in the Python world. If the design of tubes is interesting to the asyncio team, it would justify spending more time trying to integrate it

While we’re on the topic, we should also discuss forms of backpressure response that even curio doesn’t neatly handle. For example, in many systems that would like to be highly responsive it is common to want to propagate backpressure to the edge of the system and then to use that information to rapidly provide error messages to inbound work items (in HTTP speak, if your WSGI server is overloaded it would often be better to rapidly provide a 503 Service Unavailable response with a Retry-After header than to sit on the request for a potentially unbounded amount of time. The curio backpressure propagation mechanisms that Nathaniel outlined do not tolerate this situation well at all (not a slight on curio, just a natural extension of the behaviour of the socket-like APIs).

I’m interested to see what other thoughts people in this space have though. Do alternative API designs seem sensible to people?

Cory