[New-bugs-announce] [issue46843] PersistentTaskGroup API

[Joongi Kim]

New submission from Joongi Kim <me at daybreaker.info>:

I'm now tracking the recent addition and discussion of TaskGroup and cancellation scopes. It's interesting! :)

I would like to suggest to have a different mode of operation in asyncio.TaskGroup, which I named "PersistentTaskGroup".

AFAIK, TaskGroup targets to replace asyncio.gather, ensuring completion or cancellation of all tasks within the context manager scope.

I believe that a "safe" asyncio application should consist of a nested tree of task groups, which allow us to explicitly state when tasks of different purposes and contexts terminate.  For example, a task group for database transactions should be shutdown before a task group for HTTP handlers is shutdown.

To this end, in server applications with many sporadically spawned tasks throughout the whole process lifetime, there are different requirements for a task group that manages such task sets.  The tasks should *not* be cancelled upon the unhandled exceptions of sibling tasks in the task group, while we need an explicit "fallback" exception handler for those (just like "return_exceptions=True" in asyncio.gather).  The tasks belong to the task group but their references should not be kept forever to prevent memory leak (I'd suggest using weakref.WeakSet).  When terminating the task group itself, the ongoing tasks should be cancelled.  The cancellation process upon termination may happend in two phases: cancel request with initial timeout + additional limited waiting of cancellations.  (This is what Guido has mentioned in the discussion in bpo-46771.)

An initial sketch of PersistentTaskGroup is on aiotools:
https://github.com/achimnol/aiotools/blob/main/src/aiotools/ptaskgroup.py
Currently has no two-phase cancellation because it would require Python 3.11 with asyncio.Task.uncancel().

As Andrew has left a comment (https://github.com/achimnol/aiotools/issues/29#issuecomment-997437030), I think it is the time to revisit the concrete API design and whether to include PersistentTaskGroup in the stdlib or not.

----------
components: asyncio
messages: 413880
nosy: achimnol, asvetlov, gvanrossum, yselivanov
priority: normal
severity: normal
status: open
title: PersistentTaskGroup API
type: enhancement
versions: Python 3.11

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue46843>
_______________________________________