[Async-sig] Asynchronous cleanup is a problem

[Nathaniel Smith]

On Wed, Jul 6, 2016 at 5:47 PM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
>
> On Jul 6, 2016, at 5:25 PM, Yury Selivanov <yselivanov at gmail.com> wrote:
>
> The problem is that the GC can’t execute async code, and we don’t have any
> control over GC.  What if we add a mechanism to control how async generators
> (AG) are destructed.  Let’s say we add new function to the sys module -
> `sys.set_async_generator_finalizer(finalizer)`.  We already have
> sys.set_coroutine_wrapper(), so this isn’t something unprecedented.
>
>
> There isn't just one event loop though, and what trampoline to attach a
> dying coroutine to depends heavily on what event loop it came from.  It
> seems like a single global value for this in 'sys' would just be ... wrong.

I guess one could standardize a "get finalizer" protocol for coroutine
runners, where async generators (or other objects with the same
problem, I guess) would do the equivalent of

class AsyncGenerator:
    async def __anext__(self, *args, **kwargs):
        if not hasattr(self, "_finalizer"):
            self._finalizer = (yield GET_FINALIZER_SENTINEL)
        ...

    def __del__(self):
        self._finalizer.run_until_complete(self.aclose())

i.e., we provide some standard mechanism for coroutine code to take a
reference on the coroutine runner when starting up (when they do have
access to it), and then re-enter it for cleanup.

This feels a bit elaborate, though, and produces some pretty
convoluted control flow.

...does it actually work to re-enter a main loop from inside a __del__
callback? It seems like you could get into really nasty states with
multiple nested __del__ calls, or if a single sweep detects multiple
pieces of garbage with __del__ methods, then some of those __del__
calls could be delayed indefinitely while the first __del__ runs. Is
the cycle collector even re-entrant?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org