[Python-ideas] Yield-from: Suspendable generators

[Greg Ewing]

Whether you're using yield-from or not, it doesn't
seem to be possible to have a for-loop iterating
over something that is also suspendable in a
generator-based thread setting.

The basic problem is that we have one channel and
two different things we want to use it for. The
obvious answer is that we need to multiplex. Since
we're already using the entire bandwidth of the
channel (anything could be a valid yielded value)
we need to introduce some out-of-band data somehow.

Suppose we have a new expression

   suspend [<value>] [with <tag>]

This is a lot like a yield, except that it sends
a tuple (value, tag).

The existing yield expression

   yield <value>

becomes equivalent to

   suspend <value> with 'yield'

There is a new generator method to go along with
this:

   g.resume(value, tag)

If the generator is suspended at a suspend expression,
the value of the suspend expression becomes (value, tag).
If it is suspended at a yield, and the tag is 'yield'
then the value becomes the value of the yield expression.
(Not sure what to do in other cases, maybe raise an
exception.)

The existing send() method is mapped to resume() as
follows:

   def send(self, value):
     value2, tag2 = self.resume(value, 'yield')
     if tag2 == 'yield':
       return value2
     else:
       # What to do here? Ignore? Raise an exception?

So we've generalised the yield channel into a suspend
channel, which can have any number of sub-channels. We have
reserved one of these sub-channels, tagged with 'yield', for
carrying yielded values. The rest of the channels are free
for use by other things such as thread-scheduling libraries.

To complete this, we also need a variant of the for-loop
that is willing to pass values from the other channels on to
the caller. Picking a random syntax for illustration,

   for y from g(x):
     # body

would be roughly equivalent to

   it = g(x)
   value = None
   tag = 'yield'
   try:
     while 1:
       value, tag = it.resume(value, tag)
       if tag == 'yield':
         y = value
         value = None
         # body
       else:
         value, tag = suspend value with tag
   except StopIteration:
     pass

plus suitable handling of 'throw' and 'close'.

-- 
Greg