[Python-ideas] Are there asynchronous generators?

[Adam Bartoš]

Hello,

I had a generator producing pairs of values and wanted to feed all the
first members of the pairs to one consumer and all the second members to
another consumer. For example:

def pairs():
    for i in range(4):
        yield (i, i ** 2)

biconsumer(sum, list)(pairs()) -> (6, [0, 1, 4, 9])

The point is I wanted the consumers to be suspended and resumed in a
coordinated manner: The first producer is invoked, it wants the first
element. The coordinator implemented by biconsumer function invokes
pairs(), gets the first pair and yields its first member to the first
consumer. Then it wants the next element, but now it's the second
consumer's turn, so the first consumer is suspended and the second consumer
is invoked and fed with the second member of the first pair. Then the
second producer wants the next element, but it's the first consumer's turn…
and so on. In the end, when the stream of pairs is exhausted, StopIteration
is thrown to both consumers and their results are combined.

The cooperative asynchronous nature of the execution reminded me asyncio
and coroutines, so I thought that biconsumer may be implemented using them.
However, it seems that it is imposible to write an "asynchronous generator"
since the "yielding pipe" is already used for the communication with the
scheduler. And even if it was possible to make an asynchronous generator,
it is not clear how to feed it to a synchronous consumer like sum() or
list() function.

With PEP 492 the concepts of generators and coroutines were separated, so
asyncronous generators may be possible in theory. An ordinary function has
just the returning pipe – for returning the result to the caller. A
generator has also a yielding pipe – used for yielding the values during
iteration, and its return pipe is used to finish the iteration. A native
coroutine has a returning pipe – to return the result to a caller just like
an ordinary function, and also an async pipe – used for communication with
a scheduler and execution suspension. An asynchronous generator would just
have both yieling pipe and async pipe.

So my question is: was the code like the following considered? Does it make
sense? Or are there not enough uses cases for such code? I found only a
short mention in
https://www.python.org/dev/peps/pep-0492/#coroutine-generators, so possibly
these coroutine-generators are the same idea.

async def f():
    number_string = await fetch_data()
    for n in number_string.split():
        yield int(n)

async def g():
    result = async/await? sum(f())
    return result

async def h():
    the_sum = await g()

As for explanation about the execution of h() by an event loop: h is a
native coroutine called by the event loop, having both returning pipe and
async pipe. The returning pipe leads to the end of the task, the async pipe
is used for cummunication with the scheduler. Then, g() is called
asynchronously – using the await keyword means the the access to the async
pipe is given to the callee. Then g() invokes the asyncronous generator f()
and gives it the access to its async pipe, so when f() is yielding values
to sum, it can also yield a future to the scheduler via the async pipe and
suspend the whole task.

Regards, Adam Bartoš
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150624/d19e559c/attachment.html>