[Python-ideas] async/await and synchronous code (and PEP492 ?)
Koos Zevenhoven
koos.zevenhoven at aalto.fi
Wed May 6 00:23:19 CEST 2015
Hi Guido and Andrew,
Thank you for your prompt responses!
On 5.5.2015 21:48, Guido van Rossum wrote:
> Quick notes:
> - I don't think it's really possible to write realistic async code
> independently from an async framework.
And since there is asyncio in the standard library, I would assume there
typically is no reason to do that either(?) However, as a side effect of
my proposal, there would still be a way to use an if statement to pick
the right async code to match the framework, along with matching the
non-async version :).
Speaking of side effects, I think the same "__async__" variable might
also naturally provide this:
https://mail.python.org/pipermail/python-ideas/2015-April/033152.html
By the way, if I understand your first note, it might be the same as my
"The Y and L ends need to be compatible with each other for the code to
work." Sorry about the terminology. I hope the explanations of Y and L
are somewhat understandable.
> - For synchronous code that wants to use some async code, the pattern
> is simple:
> asyncio.get_event_loop().run_until_complete(some_async_call(args, etc))
> - We can probably wrap this in a convenience helper function so you
> can just write:
> asyncio.sync_wait(some_async_call(args, etc))
This is what is keeping me from using asyncio. Ignoring performance
overhead, if in any synchronous script (or interactive prompt or ipython
notebook) all calls to my library would look like that, I will happily
use my 2.7 version that uses threads. Well, I admit that the part about
"happily" is not completely true in my case.
Instead, I would be quite happy typing "await <function_call>", since
awaiting the function call (to finish/return a value) is exactly what I
would be doing, regardless of whether there is an event loop or not.
> - Note that this will fail (and rightly so!) if called when the event
> loop is already running.
>
Regarding my proposal, there would still be a way for libraries to
provide this functionality, if desired :).
Please see also the comments below.
> On Tue, May 5, 2015 at 11:00 AM, Andrew Barnert via Python-ideas
> <python-ideas at python.org <mailto:python-ideas at python.org>> wrote:
>
> It seems like it might be a lot easier to approach this from the
> other end: Is it possible to write a decorator that takes an async
> coroutine function, strips out all the awaits, and returns a
> regular sync function? If so, all you need to do is write
> everything as async, and then users can "from spam import sync as
> spam" or "from spam import async as spam" (where async just
> imports all the real functions, while sync imports them and calls
> the decorator on all of them).
>
Interesting idea. If this is possible, it would solve part of the issue,
but the "Y end" (sorry) of the chain may still need to be done by hand.
>
> That also avoids the need to have all the looking up the event
> loop, switching between different code branches, etc. inside every
> function at runtime. (Not that it matters for the performance of
> sleep(1), but it might matter for the performance of other
> functions—and, more importantly, it might make the implementation
> of those functions simpler and easier to debug through.)
>
>
This could indeed save some if statements at runtime.
Note that the if statements would not be inside every function, but only
in the ones that do the actual IO. For instance, some 3rd-party library
might use wrappers around socket send and socket recv to choose between
sync and async versions, and that might be all the IO it needs to build
several layers of async code. Even better, had someone taken the time to
provide these if statements inside the standard library, the whole
3rd-party async library would just magically work also in synchronous
code :).
Best regards,
Koos
>
> On Tuesday, May 5, 2015 7:01 AM, Koos Zevenhoven
> <koos.zevenhoven at aalto.fi <mailto:koos.zevenhoven at aalto.fi>> wrote:
>
>
>
> Hi all!
>
> I am excited about seeing what's going on with asyncio and
> PEP492 etc. I
> really like that Python is becoming more suitable for the
> increasing
> amount of async code and that the distinction between async
> functions
> and generators is increasing.
>
> In addition, however, I would also like to see the async
> functions and
> methods come even closer to regular functions and methods.
> This is
> something that is keeping me from using asyncio at the moment
> even if I
> would like to. Below I'll try to explain what and why, and a
> little bit
> of how. If it is not clear, please ask :)
>
> Motivation:
>
> One of the best things about asyncio and coroutines/async
> functions is
> that you can write asynchronous code as if it were
> synchronous, the
> difference in many places being just the use of "await"
> ("yield from")
> when calling something that may end up doing IO (somewhere
> down the
> function call chain) and that the code is run from an event loop.
>
> When writing a package that does IO, you have the option to
> make it
> either synchronous or asynchronous. Regardless of the choice,
> the code
> will look roughly the same. But what if you want to be able to
> do both?
> Should you maintain two versions, one with "async" and "await"
> everywhere and one without?
>
> Besides the keywords "async" and "await", async code of course
> differs
> from synchronous code by the functions/coroutines that are
> used for IO
> at the end of the function call chain. Here, I mean the end
> (close to)
> where the "yield" expressions are hidden in the async
> versions. At the
> other end of the calling chain, async code needs the event
> loop and
> associated framework (almost always asyncio?) which hides all
> the async
> scheduling fanciness etc. I'm not sure about the terminology,
> but I will
> use "L end" and "Y end" to refer to the two ends here. (L for
> event
> Loop; Y for Yield)
>
> The Y and L ends need to be compatible with each other for the
> code to
> work. While asyncio and the standard library might provide
> both ends in
> many cases, there can also be situations where a package would
> want to
> work with different combinations of L and Y end, or completely
> without
> an event loop, i.e. synchronously.
>
> In a very simple example, one might want to wrap different
> implementations of sleep() in a function that would pick the
> right one
> depending on the context. Perhaps something like this:
>
> async def any_sleep(seconds):
> if __async__.framework is None:
> time.sleep(1)
> elif __async__.framework is asyncio:
> await asyncio.sleep(1)
> else:
> raise RuntimeError("Was called with an unsupported
> async
> framework.")
>
> [You could of course replace sleep() with socket IO or
> whatever, but
> sleep is nice and simple. Also, a larger library would
> probably have a
> whole chain of async functions and methods before calling
> something like
> this]
>
> But if await is only allowed inside "async def", then how can
> any_sleep() be conveniently run in non-async code? Also, there is
> nothing like __async__.framework. Below, I describe what I
> think a
> potential solution might look like.
>
>
>
> Potential solution:
>
> This is simplified version; for instance, as "awaitables", I
> consider
> only async function objects here. I describe the idea in three
> parts:
>
> (1) next(...):
>
> Add a keyword argument "async_framework" (or whatever) to
> next(...) with
> a default value of None. When an async framework, typically
> asyncio,
> starts an async function object (coroutine) with a call to
> next(...), it
> would do something like next(coro, async_framework = asyncio).
> Here,
> asyncio could of course be replaced with any object that
> identifies the
> framework. This information would then be somehow attached to
> the async
> function object.
>
>
> (2) __async__.framework or something similar:
>
> Add something like __async__ that has an attribute such as
> .framework
> that allows the code inside the async function to access the
> information
> passed to next(...) by the framework (L end) using the keyword
> argument
> of next [see (1)].
>
> (3) Generalized "await":
>
> [When the world is ready:] Allow using "await" anywhere, not
> just within
> async functions. Inside async functions, the behavior of
> "await" would
> be the same as in PEP492, with the addition that it would somehow
> propagate the __async__.framework value to the awaited coroutine.
> Outside async functions, "await" would do roughly the same as
> this function:
>
> def await(async_func_obj):
> try:
> next(async_func_obj) # same as next(async_func_obj,
> async_framework = None)
> except StopIteration as si:
> return si.value
> raise RuntimeError("The function does not support
> synchronous
> execution")
>
> (This function would, of course, work in Python 3.4, but it
> would be
> mostly useless because the async functions would not know that
> they are
> being called in a 'synchronous program'. IIUC, this *function*
> would be
> valid even with PEP492, but having this as a function would be
> ugly in
> the long run.)
>
>
> Some random thoughts:
>
> With this addition to Python, one could write libraries that
> work both
> async and non-async. When await is not inside async def, one
> would
> expect it to potentially do blocking IO, just like an await
> inside async
> def would suggest that there is a yield/suspend somewhere in
> there.
>
> For testing, I tried to see if there is a reasonable way to
> make a hack
> with __async__.framework that could be set by next(), but did
> not find
> an obvious way. For instance, coro.gi_frame.f_locals is
> read-only, I
> believe.
>
> An alternative to this approach could be that await would
> implicitly
> start a temporary event loop for running the coroutine, but
> how would it
> know which event loop? This might also have a huge performance
> overhead.
>
> Relation to PEP492:
>
> This of course still needs more thinking, but I wanted to post
> it here
> now in case there is desire to prepare for something like this
> already
> in PEP492. It is not completely clear if/how this would need
> to affect
> PEP492, but some things come to mind. For example, this could
> potentially remove the need for __aenter__, __aiter__, etc. or
> even
> "async for" and "async with". If __aenter__ is defined as
> "async def",
> then a with statement would do an "await" on it, and the
> context manager
> would have __async__.framework (or whatever it would be called)
> available, for determining what behavior is appropriate.
>
> Was this clear enough to understand which problem(s) this
> would be
> solving and how? I'd be happy to hear about any thoughts on
> this :).
>
>
> Best regards,
> Koos
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <mailto:Python-ideas at python.org>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <mailto:Python-ideas at python.org>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150506/258a65d2/attachment-0001.html>
More information about the Python-ideas
mailing list