[Python-ideas] Cofunctions/yield from -> fibers

[Greg Ewing]

On 13/08/10 13:02, Ivan Pozdeev wrote:
> The original proposal for introducing 'yield from' (which is still
> in PEP 380's name) was to delegate a part of generator's work to another generator.
> However, in later discussions, the focus totally shifted to
> cooperative multitasking.

I still regard delegation as an important use case for
yield-from. I haven't dwelled much on delegation of plain
value-yielding generators in my examples because it seems
rather obvious and straightforward. My coroutine examples
are meant to show the motivation for some of the less-obvious
aspects of yield-from, such as the ability to return a value.

> In the example code Greg has given in
> http://mail.python.org/pipermail/python-ideas/2010-August/007927.html ,
>
> there's not a single use case for delegating!

That seems like a strange statement to me. There is delegation
going on all over the place in all my examples. Whenever a
generator in one of my coroutine uses yield-from to call
another one, it's delegating some of the work of that
coroutine to that generator.

The whole point of yield-from, and, to an even greater
extent, cofunctions, is to make delegation between suspendable
functions look as similar as possible to delegation between
ordinary ones.

> Instead, if it's cooperative multitasking you play with,
> switching must be independent from other activity and as explicit as possible.
> There's a technique just for that called 'fibers'
> (MS fiber API: http://msdn.microsoft.com/en-us/library/ms682661.aspx ).

I'm confused about what you're asking for. If you're
complaining that it's weird having to write 'yield from'
or 'cocall' instead of a plain function call, then I
actually agree with you. I'm trying to move cofunctions
in the direction of eliminating such call markers
completely, but I'm meeting quite a bit of resistance.

If you're saying that switching between coroutines should
involve explicitly nominating which coroutine to run next,
it would be straightforward to write a scheduler that
works this way. In fact, I can write one right now, it
would look something like this:

   def switch_to_fibre(x):
     global current_fibre
     current_fibre = x

   def main_loop():
     while 1:
       next(current_fibre)

However, it seems to me that a function such as
switch_to_fibre() is mainly useful as a primitive on
which to build higher-level scheduling strategies.
It would be possible to reformulate all of my example
schedulers to be based on such a primitive, but I'm
not sure there would be any point in doing so.
Generators already have their own primitive notion
of "run this one next", i.e. calling next() on
them, and it seems to be quite sufficient.

> `Yield from's are still valid. But only as code delegation technique, i.e. a
> shortcut to `for i in subgen(): yield i'.
> The latter statement looks brief enough for me to discard the proposal
> altogether.

But it only delegates next(), not send(), throw() or
close(). If those things are considered important for
single-function generators to have, then they are
presumably equally important for generators that are
spread over more than one function.

Also, yield-from is *much* more efficient than the
equivalent for-loop -- less than 10% of the overhead
in my current implementation.

-- 
Greg