[Async-sig] Asynchronous cleanup is a problem

[David Beazley]

> However, as far as I know curio doesn’t have the ability to schedule an operation in a synchronous manner by means of something like a Future. Is that correct? If there is no way in curio to spawn a task to occur later without having to await on it, then clearly there is no option but to allow coroutines in __aexit__ and finally: how else could curio operate?
> 

Yes, Curio does not utilize Futures or callback functions for that matter.   However, I think the real issue at hand might be much more subtle and weird.

If I understand things correctly, the goal is to implement an asynchronous generator for feeding asynchronous iteration.  This is a new kind of generator that runs inside of a coroutine (so, imagine a coroutine inside of another coroutine).   Without seeing much more, I'd guess it would look something roughly akin to this:

async def agen():
       ... some sort of async generator ...
       async yield value     # ???? Syntax ????

async def main():
       async for x in agen():
              ...

There's some kind of underlying protocol driving the async iteration, but it's *different* than what is being used by the enclosing coroutines.   Yes, there is a scheduler kernel (or event loop) that makes the outer coroutines run, but that scheduler is not driving the underlying iteration protocol of the async generator part.   So, things get weird when stuff like this happens:

async def main():
       async for x in agen():
               if x == STOP:
                     break

Here, the asynchronous generator makes it to a point, but never to completion because of the break statement.  Instead, it gets garbage collected and all hell breaks loose back in the agen() function because what happens now?  Especially if agen() uses finally or an async context manager:

async def agen():
       async with whatever:
                ...

Assuming that this is getting to the heart of the issue, I spent some time pondering it this morning and almost wonder if it could be solved by "underthinking" the solution so to speak.  For example, perhaps the __del__() method of an async-generator could just raise a RuntimeError if it's ever garbage collected before being properly terminated. Maybe you give asynchronous generators an async-close method to explicitly shut it down.  So, you'd have to do this.

async def main():
       items = agen()
       async for x in items:
             if x == STOP:
                    break
      await items.close()

Maybe the async-close() method would merely raise AsyncGeneratorExit in the generator and not enforce any other kind of semantics other than having it continue to run as a coroutine as it did before (with the understanding that the purpose of the exception is to terminate eventually). 

Since forgetting that last close() step would be easy, naturally an async generator should support the asynchronous context-management protocol.

async def main():
       async with agen() as items:
              async for x in items:
                      if x == STOP:
                            break

Perhaps the only thing missing at this point is a metaclass---or possibly a codec.  I'm not sure. 

Yikes

Cheers,
Dave