[Python-ideas] Learning from the shell in supporting asyncio background calls

[Nick Coghlan]

On 10 July 2015 at 21:51, Guido van Rossum <guido at python.org> wrote:
> As I wrote on the issue, I'm -1 on this proposal. Not only does this API
> encourage beginners to ignore the essential difference between synchronous
> functions meant to run in a thread (using synchronous I/O and pre-emptive
> CPU scheduling) and asyncio coroutines/tasks (which use overlapped I/O and
> require explicit scheduling), it also encourages avoiding the "await"
> primitive (formerly "yield from") in favor of a function call which cannot
> be used from within a coroutine/task.

My apologies for the confusion - the revised proposal focuses on
coroutines, not threads. With the benefit of hindight, leaving the
implementation details out of the python-ideas post was clearly a
mistake, as my previous posts had been more focused on threads. I've
added the full implementation details in my reply to Oscar, which will
hopefully make the revised proposal clearer.

The blog post goes into detail on this - it specifically takes a
synchronous function, replaces it with an asynchronous coroutine using
the await syntax, and then uses run_in_background() to manipulate the
asynchronous version from the REPL.

The main operation I use with "run_in_foreground" in the post is
actually asyncio.sleep, as "run_in_foreground(asyncio.sleep(0))" was
the simplest way I found to single step the event loop, and it also
allows you to trivially say "run the event loop for 5 seconds", etc.
Concatenating some of the example code from the post together gives
this demonstration of the basic UX:

    >>> async def ticker():
    ...     for i in itertools.count():
    ...         print(i)
    ...         await asyncio.sleep(1)
    ...
    >>> ticker1 = run_in_background(ticker())
    >>> ticker1
    <Task pending coro=<ticker() running at <stdin>:1>>
    >>> run_in_foreground(asyncio.sleep(5))
    0
    1
    2
    3
    4

If there isn't a coroutine currently running in the foreground, then
background coroutines don't run either. All of the currently running
tasks can be interrogated through the existing
asyncio.Task.all_tasks() class method.

> This particular spelling moreover introduces a "similarity" between
> foreground and background tasks that doesn't actually exist.

The concept behind the revised proposal is layering the simpler
foreground/background task representational model on top of the full
complexity of the asyncio implementation model.

The "run_in_foreground" naming is technically a lie - what actually
gets run in the foreground is the current thread's event loop.
However, I think it's an acceptable and useful lie, as what it does is
run the event loop in the current thread until the supplied future
produces a result, which means the current thread isn't going to be
doing anything other than running the event loop until the specified
operation is completed.

This approach *doesn't* expose the full power of asyncio and native
coroutines, but it exposes a lot of it, and it should be relatively
easy to grasp for anyone that's already familiar with background
processes in POSIX shell environments.

> The example suggests that this should really be a pair of convenience
> functions in collections.futures, as it does not make any use of asyncio.

While that was true of the previous proposal (which always used the
executor), this new proposal only falls back to using run_in_executor
if asyncio.ensure_future fails with TypeError and the supplied
background task target is a callable.

Regards,
Nick.

P.S. If anyone reading this isn't already familiar with the concept of
representational models vs implementation models, then I highly
recommend http://www.uxpassion.com/blog/implementation-mental-representation-models-ux-user-experience/
as a good introduction to the idea

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia