[Python-Dev] API design: where to add async variants of existing stdlib APIs?

[Nathaniel Smith]

On Tue, Feb 28, 2017 at 9:42 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Short version:
>
> - there are some reasonable requests for async variants of contextlib APIs
> for 3.7
> - prompted by Raymond, I'm thinking it actually makes more sense to add
> these in a new `asyncio.contextlib` module than it does to add them directly
> to the existing module
> - would anyone object strongly to my asking authors of the affected PRs to
> take their changes in that direction?

IMHO this is a good idea *iff* the new APIs really are bound to
asyncio, rather than being generic across all uses of async/await.

It sounds like that's not the case, though? There are definitely use
cases for acontextmanager in programs that don't use asyncio at all
(but rather use twisted, curio, ...). Guido's even suggested that he'd
like to see a PEP for an "asyncio2" within the 3.7/3.8 timeframe:
https://mail.python.org/pipermail/async-sig/2016-November/000175.html

asyncio is an important use case for async/await, but it's definitely
not the only one. In cases where it's possible to write generic
machinery in terms of async/await semantics, without assuming any
particular coroutine runner's semantics, then I strongly urge you to
do so.

> Longer version:
>
> There are a couple of open issues requesting async variants of some
> contextlib APIs (asynccontextmanager and AsyncExitStack). I'm inclined to
> accept both of them, but Raymond raised a good question regarding our
> general design philosophy for these kinds of additions: would it make more
> sense to put these in an "asyncio.contextlib" module than it would to add
> them directly to contextlib itself?
>
> The main advantage I see to the idea is that if someone proposed adding an
> "asyncio" dependency to contextlib, I'd say no. For the existing
> asynccontextmanager PR, I even said no to adding that dependency to the
> standard contextlib test suite, and instead asked that the new tests be
> moved out to a separate file, so the existing tests could continue to run
> even if asyncio was unavailable for some reason.

asyncio is a stable, non-provisional part of the standard library;
it's not going anywhere. Personally I wouldn't be bothered about
depending on it for tests. (My async_generator library is in a similar
position: it isn't tied to any particular framework, and I don't even
use asyncio myself, but the test suite depends on asyncio because hey,
whatever, everyone already has it and it plays the role of generic
coroutine runner as well as anything else does.)

OTOH if you don't need to do any I/O then it's actually pretty easy to
write a trivial coroutine runner. I think something like this should
be sufficient to write any test you might want:

@types.coroutine
def send_me(value):
    return yield ("value", value)

@types.coroutine
def throw_me(exc):
    yield ("error", exc)

async def yield_briefly():
    await send_me(None)

def run(async_fn, *args, **kwargs):
    coro = async_fn(*args, **kwargs)
    next_msg = ("value", None)
    try:
        while True:
            if next_msg[0] == "value":
                next_msg = coro.send(next_msg[1])
            else:
                next_msg = coro.throw(next_msg[1])
    except StopIteration as exc:
        return exc.value

> While rejecting the idea of an asyncio dependency isn't a problem for
> asyncontextmanager specifically (it's low level enough for it not to
> matter), it's likely to be more of a concern for the AsyncExitStack API,
> where the "asyncio.iscoroutinefunction" introspection API is likely to be
> quite helpful, as are other APIs like `asyncio.ensure_future()`.

FYI FWIW, every time I've tried to use iscoroutinefunction so far I've
ended up regretting it and ripping it out again :-). The problem is
that people will do things like apply a decorator to a coroutine
function, and get a wrapped function that returns a coroutine object
and which is interchangeable with a real coroutine function in every
way except that iscoroutinefunction returns False. And there's no
collections.abc.CoroutineFunction (not sure how that would even work).
Better to just call the thing and then do an isinstance(...,
collections.abc.Coroutine) on the return value.

I haven't had a reason to try porting ExitStack to handle async
context managers yet, so I can't speak to it beyond that :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org