[Python-ideas] async/await and synchronous code (and PEP492 ?)

[Guido van Rossum]

Quick notes:
- I don't think it's really possible to write realistic async code
independently from an async framework.
- For synchronous code that wants to use some async code, the pattern is
simple:
    asyncio.get_event_loop().run_until_complete(some_async_call(args, etc))
- We can probably wrap this in a convenience helper function so you can
just write:
    asyncio.sync_wait(some_async_call(args, etc))
- Note that this will fail (and rightly so!) if called when the event loop
is already running.

On Tue, May 5, 2015 at 11:00 AM, Andrew Barnert via Python-ideas <
python-ideas at python.org> wrote:

> It seems like it might be a lot easier to approach this from the other
> end: Is it possible to write a decorator that takes an async coroutine
> function, strips out all the awaits, and returns a regular sync function?
> If so, all you need to do is write everything as async, and then users can
> "from spam import sync as spam" or "from spam import async as spam" (where
> async just imports all the real functions, while sync imports them and
> calls the decorator on all of them).
>
> That also avoids the need to have all the looking up the event loop,
> switching between different code branches, etc. inside every function at
> runtime. (Not that it matters for the performance of sleep(1), but it might
> matter for the performance of other functions—and, more importantly, it
> might make the implementation of those functions simpler and easier to
> debug through.)
>
>
>
>   On Tuesday, May 5, 2015 7:01 AM, Koos Zevenhoven <
> koos.zevenhoven at aalto.fi> wrote:
>
>
>
> Hi all!
>
> I am excited about seeing what's going on with asyncio and PEP492 etc. I
> really like that Python is becoming more suitable for the increasing
> amount of async code and that the distinction between async functions
> and generators is increasing.
>
> In addition, however, I would also like to see the async functions and
> methods come even closer to regular functions and methods. This is
> something that is keeping me from using asyncio at the moment even if I
> would like to. Below I'll try to explain what and why, and a little bit
> of how. If it is not clear, please ask :)
>
> Motivation:
>
> One of the best things about asyncio and coroutines/async functions is
> that you can write asynchronous code as if it were synchronous, the
> difference in many places being just the use of "await" ("yield from")
> when calling something that may end up doing IO (somewhere down the
> function call chain) and that the code is run from an event loop.
>
> When writing a package that does IO, you have the option to make it
> either synchronous or asynchronous. Regardless of the choice, the code
> will look roughly the same. But what if you want to be able to do both?
> Should you maintain two versions, one with "async" and "await"
> everywhere and one without?
>
> Besides the keywords "async" and "await", async code of course differs
> from synchronous code by the functions/coroutines that are used for IO
> at the end of the function call chain. Here, I mean the end (close to)
> where the "yield" expressions are hidden in the async versions. At the
> other end of the calling chain, async code needs the event loop and
> associated framework (almost always asyncio?) which hides all the async
> scheduling fanciness etc. I'm not sure about the terminology, but I will
> use "L end" and "Y end" to refer to the two ends here. (L for event
> Loop; Y for Yield)
>
> The Y and L ends need to be compatible with each other for the code to
> work. While asyncio and the standard library might provide both ends in
> many cases, there can also be situations where a package would want to
> work with different combinations of L and Y end, or completely without
> an event loop, i.e. synchronously.
>
> In a very simple example, one might want to wrap different
> implementations of sleep() in a function that would pick the right one
> depending on the context. Perhaps something like this:
>
>   async def any_sleep(seconds):
>       if __async__.framework is None:
>           time.sleep(1)
>       elif __async__.framework is asyncio:
>           await asyncio.sleep(1)
>       else:
>           raise RuntimeError("Was called with an unsupported async
> framework.")
>
> [You could of course replace sleep() with socket IO or whatever, but
> sleep is nice and simple. Also, a larger library would probably have a
> whole chain of async functions and methods before calling something like
> this]
>
> But if await is only allowed inside "async def", then how can
> any_sleep() be conveniently run in non-async code? Also, there is
> nothing like __async__.framework. Below, I describe what I think a
> potential solution might look like.
>
>
>
> Potential solution:
>
> This is simplified version; for instance, as "awaitables", I consider
> only async function objects here. I describe the idea in three parts:
>
> (1) next(...):
>
> Add a keyword argument "async_framework" (or whatever) to next(...) with
> a default value of None. When an async framework, typically asyncio,
> starts an async function object (coroutine) with a call to next(...), it
> would do something like next(coro, async_framework = asyncio). Here,
> asyncio could of course be replaced with any object that identifies the
> framework. This information would then be somehow attached to the async
> function object.
>
>
> (2) __async__.framework or something similar:
>
> Add something like __async__ that has an attribute such as .framework
> that allows the code inside the async function to access the information
> passed to next(...) by the framework (L end) using the keyword argument
> of next [see (1)].
>
> (3) Generalized "await":
>
> [When the world is ready:] Allow using "await" anywhere, not just within
> async functions. Inside async functions, the behavior of "await" would
> be the same as in PEP492, with the addition that it would somehow
> propagate the __async__.framework value to the awaited coroutine.
> Outside async functions, "await" would do roughly the same as this
> function:
>
>   def await(async_func_obj):
>       try:
>           next(async_func_obj)  # same as next(async_func_obj,
> async_framework = None)
>       except StopIteration as si:
>           return si.value
>       raise RuntimeError("The function does not support synchronous
> execution")
>
> (This function would, of course, work in Python 3.4, but it would be
> mostly useless because the async functions would not know that they are
> being called in a 'synchronous program'. IIUC, this *function* would be
> valid even with PEP492, but having this as a function would be ugly in
> the long run.)
>
>
> Some random thoughts:
>
> With this addition to Python, one could write libraries that work both
> async and non-async. When await is not inside async def, one would
> expect it to potentially do blocking IO, just like an await inside async
> def would suggest that there is a yield/suspend somewhere in there.
>
> For testing, I tried to see if there is a reasonable way to make a hack
> with __async__.framework that could be set by next(), but did not find
> an obvious way. For instance, coro.gi_frame.f_locals is read-only, I
> believe.
>
> An alternative to this approach could be that await would implicitly
> start a temporary event loop for running the coroutine, but how would it
> know which event loop? This might also have a huge performance overhead.
>
> Relation to PEP492:
>
> This of course still needs more thinking, but I wanted to post it here
> now in case there is desire to prepare for something like this already
> in PEP492. It is not completely clear if/how this would need to affect
> PEP492, but some things come to mind. For example, this could
> potentially remove the need for __aenter__, __aiter__, etc. or even
> "async for" and "async with". If __aenter__ is defined as "async def",
> then a with statement would do an "await" on it, and the context manager
> would have __async__.framework (or whatever it would be called)
> available, for determining what behavior is appropriate.
>
> Was this clear enough to understand which problem(s) this would be
> solving and how? I'd be happy to hear about any thoughts on this :).
>
>
> Best regards,
> Koos
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150505/6e535bc8/attachment.html>