[Python-ideas] Learning from the shell in supporting asyncio background calls

[Guido van Rossum]

As I wrote on the issue, I'm -1 on this proposal. Not only does this API
encourage beginners to ignore the essential difference between synchronous
functions meant to run in a thread (using synchronous I/O and pre-emptive
CPU scheduling) and asyncio coroutines/tasks (which use overlapped I/O and
require explicit scheduling), it also encourages avoiding the "await"
primitive (formerly "yield from") in favor of a function call which cannot
be used from within a coroutine/task.

This particular spelling moreover introduces a "similarity" between
foreground and background tasks that doesn't actually exist.

The example suggests that this should really be a pair of convenience
functions in collections.futures, as it does not make any use of asyncio.

On Fri, Jul 10, 2015 at 12:49 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Hi folks,
>
> Based on the recent discussions Sven kicked off regarding the
> complexity of interacting with asyncio from otherwise synchronous
> code, I came up with an API design that I like inspired by the way
> background and foreground tasks in the POSIX shell work.
>
> My blog post about this design is at
>
> http://www.curiousefficiency.org/posts/2015/07/asyncio-background-calls.html
> ,
> but the essential components are the following two APIs:
>
>     def run_in_background(target, *, loop=None):
>         """Schedules target as a background task
>
>         Returns the scheduled task.
>
>         If target is a future or coroutine, equivalent to
> asyncio.ensure_future
>         If target is a callable, it is scheduled in the default executor
>         """
>         ...
>
>     def run_in_foreground(task, *, loop=None):
>         """Runs event loop in current thread until the given task completes
>
>         Returns the result of the task.
>         For more complex conditions, combine with asyncio.wait()
>         To include a timeout, combine with asyncio.wait_for()
>         """
>         ...
>
> run_in_background is akin to invoking a shell command with a trailing
> "&" - it puts the operation into the background, leaving the current
> thread to move on to the next operation (or wait for input at the
> REPL). When coroutines are scheduled, they won't start running until
> you start a foreground task, while callables delegated to the default
> executor will start running immediately.
>
> To actually get the *results* of that task, you have to run it in the
> foreground of the current thread using run_in_foreground - this is
> akin to bringing a background process to the foreground of a shell
> session using "fg".
>
> To relate this idea back to some of the examples Sven was discussing,
> here's how translating some old serialised synchronous code to use
> those APIs might look in practice:
>
>     # Serial synchronous data loading
>     def load_and_process_data():
>         data1 = load_remote_data_set1()
>         data2 = load_remote_data_set2()
>         return process_data(data1, data2)
>
>     # Parallel asynchronous data loading
>     def load_and_process_data():
>         future1 = asyncio.run_in_background(load_remote_data_set1_async())
>         future2 = asyncio.run_in_background(load_remote_data_set2_async())
>         data1 = asyncio.run_in_foreground(future1)
>         data2 = asyncio.run_in_foreground(future2)
>         return process_data(data1, data2)
>
> The application remains fundamentally synchronous, but the asyncio
> event loop is exploited to obtain some local concurrency in waiting
> for client IO operations.
>
> Regards,
> Nick.
>
> P.S. time.sleep() and asyncio.sleep() are rather handy as standins for
> blocking and non-blocking IO operations. I wish I'd remembered that
> earlier :)
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150710/83565dd1/attachment.html>