[Python-ideas] Revised revised revised PEP on yield-from

[Guido van Rossum]

On Fri, Feb 13, 2009 at 5:31 PM, Raymond Hettinger <python at rcn.com> wrote:
> Are there any use cases that warrant all this complexity?
> I not yet seen a single piece of real-world code that would
> benefit from yield-from having pass-throughs for send/throw/close.
> So far, this seems to have been a purely theoretical exercise
> what is possible, but it doesn't seem to include investigation as to
> whether it is actually useful.
> In the absence of real-world use cases, it might still be helpful
> to look at some contrived, hypothetical use cases so we can
> see if the super-powered version actually provides a better
> solution (is the code more self-evidently correct, is the construct
> easy to use and understand, is it awkward to use)?
>
> The proto-pep seems heavy on specification and light on
> showing that this is actually something we want to have.
> Plenty of folks have shown an interest in a basic version
> of yield-every or yield-from, but prior to this protoPEP,
> I've never seen any request for or discussion of a version
> that does pass-throughs for send/throw/close.

While I haven't read the PEP thouroughly, I believe I understand the
concept of pass-through and I think I have a compelling use case, at
least for passing through .send(). The rest then shouldn't be a
problem. Let's also not forget that 99% of all uses of generators
don't involve .send(), .throw() or .close().

My use case is flattening of trees, for example parse trees. For
concreteness, assume a node has a label and a list of children. The
iteration should receive ENTER and LEAVE pseudo-labels when entering a
level. We can then write a pre-order iterator like this, using
yield-from without caring about pass-through:

def __iter__(self):
  yield self.label
  if self.children:
    yield ENTER
    for child in self.children:
      yield from child
    yield LEAVE

Now suppose the caller of the iteration wants to be able to
occasionally truncate the traversal, e.g. it may not be interested in
the subtree for certain labels, or it may want to skip very deep
trees. It's not possible to anticipate what the caller is wants to
truncate, so we don't want to build direct support for e.g. skip-lists
or level-control into the iterator. Instead, the caller now uses
.send(SKIP) when it wants to skip a subtree. The iterator responds
with a SKIPPED pseudo-label. For example:

def __iter__(self):
  skip = yield self.label
  if skip == SKIP:
    yield SKIPPED
  else:
    skip = yield ENTER
    if skip == SKIP:
      yield SKIPPED
    else:
      for child in self.children:
        yield from child
    yield LEAVE

I believe the pass-through semantics proposed for yield-from are
*exactly* what we need in this case. Without it, the for-loop would
have to be written like this:

for child in self.children:
  it = iter(child)
  while True:
    try:
      value = it.send(skip)
    except StopIteration:
      break
    skip = yield value

Other remarks:

(a) I don't know if the PEP proposes that "yield from expr" should
return the last value returned by (i.e. sent to) a yield somewhere
deeply nested; I think this would be useful.

(b) I hope the PEP also explains what to do if "expr" is not a
generator but some other kind of iterator. IMO it should work as long
as .send() etc. are not used. I think it would probably be safest to
raise an exception is .send() is used and the receiving iterator is
not a generator. For .throw() and .close() it would probably be most
useful to let them have their effect in the last generator on the
stack.

(c) A quick skim of the PEP didn't show suggestions for how to
implement this. I think this needs to be addressed. I don't think it
will be possible to literally replace the outer generator with the
inner one while that is running; the treatment of StopIteration
probably requires some kind of chaining, so that there is still a cost
associated with deeply nested yield-from clauses. However it could be
much more efficient than explicit for-loops.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)