Asynchronous programming

[Paul Rubin]

Steven D'Aprano <steve+python at pearwood.info> writes:
> Is there a good beginner's tutorial introducing the basics of asynchronous
> programming? Starting with, why and where would you use it?

You might look at some node.js tutorials since there are about a
gazillion of them and some of them must be good.  Also twistedmatrix.org
for an older Python package of that sort.

The basic reason to do it is it lets you serve a lot of concurrent i/o
channels (network connections, say) without using threads.  If you want
to read a packet, you launch a non-blocking read that returns
immediately, and then transfer control (often through some
behind-the-scenes magic) to an event loop that dispatches back to you
after your read request completes, through either a callback or a
coroutine jump depending on which framework you're using.  In Python
that gets you much higher performance than threads, plus you avoid the
usual bunch of hazards that parents used to scare their kids with about
multi-threaded programming.

> How is this the same as, or different from, event-based programming? 

The basic characteristic of asynchronous programming is that it involves
changing all your usual blocking i/o calls to non-blocking ones, so your
program can keep running as soon as your request is started.  That's not
always the case with event-based programming in general, where event
handlers might do i/o.  The basic problem you face is the inversion of
control needed to get in and out of the event loop, compared to the
sequential blocking i/o that you'd use in a threaded or non-concurrent
program.

The new thing in Python is asyncio, which uses coroutines instead of
callbacks to switch between event handlers.  It's asynchronous
programming either way, but coroutines can be seen as a cleaner
implementation technique depending on your POV.  Coroutines in the form
of "yield from" first appeared in Python 3.??? and became more fully
integrated with the new async and await keywords.  Before that, async
concurrency in Python was done using callback schemes.  Twisted used to
work like that (I don't know if it still does).  I hated it but I was in
a minority.

> If I'm using async and await in Python, where's the event loop? What
> are the messages, and where are they sent? Or am I on the wrong track
> altogether?

It's supplied by the asyncio package.  Btw, David Beazely's article
about Python coroutines is very good, though it was written before the
new keywords were added.