[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Guido van Rossum]

On Fri, Oct 19, 2012 at 7:41 PM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> I'm not entirely sure whether I'm hijacking the thread here... I have to admit I've somewhat lost track with all the renames. The discussion has been very interesting (I really like the 'codef' idea, and decorators can provide this without requiring syntax changes) regardless of which thread is active.
>
> I have spent a bit of time writing up the approach that we (Dino, who posted it here originally, myself and with some advice from colleagues who are working on a similar API for C++) came up with and implemented.
>
> I must apologise for the length - I got a little carried away with background information, but I do believe that it is important for us to understand exactly what problem we're trying to solve so that we aren't distracted by "new toys".
>
> The write-up is here: http://stevedower.id.au/blog/async-api-for-python/
>
> I included code, since there have been a few people asking for prototype implementations, so if you want to skip ahead to the code (which is quite heavily annotated) it is at http://stevedower.id.au/blog/async-api-for-python/#thecode or http://stevedower.id.au/downloads/PythonAsync.zip (I based my example on Greg's socket spam, so thanks for that!)
>
> And no, I'm not collecting any ad revenue from the page, so feel free to visit as often as you like and use up my bandwidth.
>
> Let the tearing apart of my approach commence! :)

Couple of questions and comments.

- You mention a query interface a few times but there are no details
in your example code; can you elaborate? (Or was that a typo for
queue?)

- This is almost completely isomorphic with NDB's tasklets, except
that you borrow the Future class implementation from
concurrent.futures -- I think that's the wrong building block to start
with, because it is linked too closely to threads.

- There is a big speed difference between yield from <generator> and
yield <future>. With yield <future>, the scheduler has to do
significant work for each yield at an intermediate level, whereas with
yield from, the schedule is only involved when actual blocking needs
to be performed. In my experience, real code has lots of intermediate
levels. Therefore I would like to use yield from. You can already do
most things with yield from that you can do with Futures; there are a
few operations that need a helper (in particular spawning truly
concurrent tasks), but the helper code can be much simpler than the
Future object, and isn't needed as often, so it's still a bare win.

- Nit: I don't like calling the event loop context; there are too many
things called context (e.g. context managers in Python), so I prefer
to call it what it is -- event loop or I/O loop.

- Nittier nit: you have a few stray colons, e.g. "it = iter(fn(*args,
**kwargs)):" .

-- 
--Guido van Rossum (python.org/~guido)