[Python-ideas] Possible PEP 380 tweak

[Nick Coghlan]

On Sat, Oct 30, 2010 at 11:15 AM, Guido van Rossum <guido at python.org> wrote:
> BTW I don't think I like piggybacking a return value on GeneratorExit.
> Before you know it people will be writing except blocks catching
> GeneratorExit intending to catch one coming from inside but
> accidentally including a yield in the try block and catching one
> coming from the outside. The nice thing about how GeneratorExit works
> today is that you needn't worry about it coming from inside, since it
> always comes from the outside *first*. This means that if you catch a
> GeneratorExit, it is either one you threw into a generator yourself
> (it just bounced back, meaning the generator didn't handle it at all),
> or one that was thrown into you. But the pattern of catching
> GeneratorExit and responding by returning a value is a reasonable
> extension of the pattern of catching GeneratorExit and doing other
> cleanup.

(TLDR version: I'm -1 on Guido's modified close() semantics if there
is no way to get the result out of a yield from expression that is
terminated by GeneratorExit, but I'm +1 if we tweak PEP 380 to make
the result available on the reraised GeneratorExit instance, thus
allowing framework authors to develop ways to correctly unwind a
generator stack in response to close())

Stepping back a bit, let's look at the ways a framework may "close" a
generator-based operation (or substep of a generator).

1. Send in a sentinel value (often None, but you could easily reuse
the exception types as sentinel values  as well)
2. Throw in GeneratorExit explicitly
3. Throw in StopIteration explicitly
4. Throw in a different specific exception
5. Call g.close()

Having close() return a value only helps with the last option, and
only if the coroutine is set up to work that way. Yield from also
isn't innately set up to unwind correctly in any of these cases,
without some form of framework based signalling from the inner
generator to indicate whether or not the outer generator should
continue or bail out.

Now, *if* close() were set up to return a value, then that second
point makes the idea less useful than it appears. To go back to the
simple summing example (not my
too-complicated-for-a-mailing-list-discussion version which I'm not
going to try to rehabilitate):

def gtally():
  count = tally = 0
  try:
    while 1:
      tally += yield
      count += 1
  except GeneratorExit:
    pass
  return count, tally

Fairly straightforward, but one of the promises of PEP 380 is that it
allows us to factor out some or all of a generator's internal logic
without affecting the externally visible semantics. So, let's do that:

  def gtally2():
    return (yield from gtally())

Unless the PEP 380 yield from expansion is changed, Guido's proposed
"close() returns the value on StopIteration" just broke this
equivalence for gtally2() - since the yield from expansion turns the
StopIteration back into a GeneratorExit, the return value of
gtally2.close is always going to be None instead of the expected
(count, tally) 2-tuple. Since the value of the internal call to
close() is thrown away completely, there is absolute nothing the
author of gtally2() can do to fix it (aside from not using yield from
at all). To me, if Guido's idea is adopted, this outcome is as
illogical and unacceptacle as the following returning None:

  def sum2(seq):
    return sum(seq)

We already thrashed out long ago that the yield from handling of
GeneratorExit needs to work the way it does in order to serve its
primary purpose of releasing resources, so allowing the inner
StopIteration to propagate with the exception value attached is not an
option.

The question is whether or not there is a way to implement the
return-value-from-close() idiom in a way that *doesn't* completely
break the equivalence between gtally() and gtally2() above. I think
there is: store the prospective return-value on the GeneratorExit
instance and have the yield from expansion provide the most recent
return value as it unwinds the stack.

To avoid giving false impressions as to which level of the stack
return values are from, gtally2() would need to be implemented a bit
differently in order to *also* convert GeneratorExit to StopIteration:

  def gtally2():
    # The PEP 380 equivalent of a "tail call" if g.close() returns a value
    try:
      yield from gtally()
    except GeneratorExit as ex:
      return ex.value

Specific proposed additions/modifications to PEP 380:

1. The new "value" attribute is added to GeneratorExit as well as
StopIteration and is explicitly read/write

2. The semantics of the generator close method are modified to be:

  def close(self):
    try:
      self.throw(GeneratorExit)
    except StopIteration as ex:
      return ex.value
    except GeneratorExit:
      return None # Ignore the value, as it didn't come from the
outermost generator
    raise RuntimeError("Generator ignored GeneratorExit")

3.  The GeneratorExit handling semantics for the yield from expansion
are modified to be:

        except GeneratorExit as _e:
            try:
                _m = _i.close
            except AttributeError:
                pass
            else:
                _e.value = _m() # Store close() result on the exception
            raise _e

With these modifications, a framework could then quite easily provide
a context manager to make the idiom a little more readable and hide
the fact that GeneratorExit is being caught at all:

class GenResult():
    def __init__(self): self.value = None

@contextmanager
def generator_return():
    result = GenResult()
    try:
      yield
    except GeneratorExit as ex:
      result.value = ex.value

def gtally():
  # The CM suppresses GeneratorExit, allowing us
  # to convert it to StopIteration
  count = tally = 0
  with generator_return():
    while 1:
      tally += yield
      count += 1
  return count, tally

def gtally2():
  # The CM *also* collects the value of any inner
  # yield from expression, allowing easier tail calls
  with generator_return() as result:
    yield from gtally()
  return result.value

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia