[Python-Dev] PEP 380 (yield from a subgenerator) comments

[Nick Coghlan]

Antoine Pitrou wrote:
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>> And that it formally expanded to:
>>
>> <snip 20 lines of code with multiple try/except/finally clauses and various
> conditionals>
> 
> Do we really want to add a syntactic feature which has such a complicated
> expansion? I fear it will make code using "yield from" much more difficult to
> understand and audit.

Yes, I think we do. The previous argument against explicit syntactic
support for invoking subiterators was that it was trivial to do so by
iterating over the subiterator and yielding each item in turn.

With the additional generator features introduced by PEP 342, that is no
longer the case: as described in Greg's PEP, simple iteration doesn't
support send() and throw() correctly. The gymnastics needed to support
send() and throw() actually aren't that complex when you break them
down, but they aren't trivial either.

Whether or not different people will find code using "yield from"
difficult to understand or not will have more to do with their grasp of
the concepts of cooperative multitasking in general more so than the
underlying trickery involved in allowing truly nested generators.

Here's an annotated version of the expansion that will hopefully make
things clearer:

  # Create the subiterator
  _i = iter(EXPR)
  # Outer try block serves two purposes:
  #  - retrieve expression result from StopIteration instance
  #  - ensure _i.close() is called if it exists
  try:
      # Get first value to be yielded
      _u = _i.next()
      while 1:
          # Inner try block allows exceptions passed in via
          # the generator's throw() method to be passed to
          # the subiterator
          try:
              _v = yield _u
          except Exception, _e:
              # An exception was thrown into this
              # generator. If the subiterator has
              # a throw() method, then we pass the
              # exception down. Otherwise, we
              # propagate the exception in the
              # current generator
              # Note that SystemExit and
              # GeneratorExit are never passed down.
              # For those, we rely on the close()
              # call in the outer finally block
              _m = getattr(_i, 'throw', None)
              if _m is not None:
                  # throw() will either yield
                  # a new value, raise StopIteration
                  # or reraise the original exception
                  _u = _m(_e)
              else:
                  raise
          else:
              if _v is None:
                  # Get the next subiterator value
                  _u = _i.next()
              else:
                  # A value was passed in using
                  # send(), so attempt to pass it
                  # down to the subiterator.
                  # AttributeError will be raised
                  # if the subiterator doesn't
                  # provide a send() method
                  _u = _i.send(_v)
  except StopIteration, _e:
      # Subiterator ended, get the expression result
      _expr_result = _e.value
  finally:
      # Ensure close() is called if it exists
      _m = getattr(_i, 'close', None)
      if _m is not None:
          _m()
  RESULT = _expr_result


On further reflection (and after reading a couple more posts on
python-ideas relating to this PEP), I have two more questions/concerns:

1. The inner try/except is completely pointless if the subiterator
doesn't have a throw() method. Would it make sense to have two versions
of the inner loop (with and without the try block) and choose which one
to use based on whether or not the subiterator has a throw() method?
(Probably not, since this PEP is mainly about generators as cooperative
pseudo-threads and in such situations all iterators involved are likely
to be generators and hence have throw() methods. However, I think the
question is at least worth thinking about.)

2. Due to a couple of bug reports against 2.5,
contextlib.GeneratorContextManager now takes extra care when handling
exceptions to avoid accidentally suppressing explicitly thrown in
StopIteration instances. However, the current expansion in PEP 380
doesn't check if the StopIteration caught by the outer try statement was
one that was originally thrown into the generator rather than an
indicator that the subiterator naturally reached the end of its
execution. That isn't a difficult behaviour to eliminate, but it does
require a slight change to the semantic definition of the new expression:

  _i = iter(EXPR)
  _thrown_exc = None
  try:
      _u = _i.next()
      while 1:
          try:
              _v = yield _u
          except Exception, _e:
              _thrown_exc = _e
              _m = getattr(_i, 'throw', None)
              if _m is not None:
                  _u = _m(_e)
              else:
                  raise
          else:
              if _v is None:
                  _u = _i.next()
              else:
                  _u = _i.send(_v)
  except StopIteration, _e:
      if _e is _thrown_exc:
          # Don't suppress StopIteration if it
          # was thrown in from outside the
          # generator
          raise
      _expr_result = _e.value
  finally:
      _m = getattr(_i, 'close', None)
      if _m is not None:
          _m()
  RESULT = _expr_result

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------