[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Sun, Nov 26, 2017 at 6:51 PM, Guido van Rossum <guido at python.org> wrote:
> On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith <njs at pobox.com> wrote:
[..]
>> - async/await has associated thread-global state like
>> sys.set_coroutine_wrapper and sys.set_asyncgen_hooks. Generally async
>> libraries assume that they own these, and arbitrarily weird things may
>> happen if you have multiple async/await coroutine runners in same
>> thread with no coordination between them.
>
>
> The existence of these is indeed a bit unfortunate for this use case. I'm
> CC'ing Yury to ask him if he can think of a different way to deal with the
> problems that these are supposed to solve. For each, the reason they exist
> is itself an edge case -- debugging for the former, finalization for the
> latter. A better solution for these problem may also be important for
> situations where multiple event loops exist (in the same thread, e.g.
> running alternately). Maybe a context manager could be used to manage this
> state better?

Yeah, both of them are for solving edge cases:

- sys.set_coroutine_wrapper() is a debug API: asyncio uses it to
slightly enhance the warning about non-awaited coroutines by showing
where they were *created*.  Capturing traceback when we create every
coroutine is expensive, hence we only do that in asyncio debug mode.

- sys.set_asyncgen_hooks() is more important, we use it to let event
loops finalize partially iterated and then abandoned asynchronous
generators.

The rule of thumb is to get the previous coro-wrapper/asyncgen-hook,
set your own (both are thread specific), and after you're done --
restore the saved old versions back.  This should work fine in all use
cases (even a trio event loop nested in an asyncio event loop).  But
nested coroutine loops is a sign of bad design, the nested loop will
completely block the execution of the outer event loop, so situations
like that should be avoided.  Not to mention that debugging becomes
much harder when you have complex nested coroutine runners.

If someone wants to run some async/await code without an event loop
for educational purposes or to implement some pattern, they likely
don't even need to use any of the above APIs.

For me the status quo is OK: asyncio/twisted/curio/trio use these APIs
internally, and for them it shouldn't be a problem to use a couple
low-level functions from sys.

That said I have a couple ideas:

- We can simplify these low-level APIs by combining
set_coroutine_wrapper() and set_asyncgen_hooks() into one function --
set_async_hooks(coroutine_wrapper=, firstiter=, etc). It could return
a context-manager, so the following would be possible:

   with sys.set_async_hooks(...):
       # run async code or an event loop

- Another option is to design an API that would let you to stack your
coro-wrapper/asyncgen-hooks, so that many coroutine runners can
control coroutines/generators simultaneously.  This sounds very
complex to me though, and I haven't seen any compelling real-world use
case that would require this kind of design.

sys.set_coroutine_wrapper() is marked as an experimental debug API;
sys.set_asyncgen_hooks() is still provisional.  So we can in theory
change/improve both of them.

Yury