[Python-ideas] Possible PEP 380 tweak

[Guido van Rossum]

On Fri, Oct 29, 2010 at 8:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Oct 30, 2010 at 11:15 AM, Guido van Rossum <guido at python.org> wrote:
>> BTW I don't think I like piggybacking a return value on GeneratorExit.
>> Before you know it people will be writing except blocks catching
>> GeneratorExit intending to catch one coming from inside but
>> accidentally including a yield in the try block and catching one
>> coming from the outside. The nice thing about how GeneratorExit works
>> today is that you needn't worry about it coming from inside, since it
>> always comes from the outside *first*. This means that if you catch a
>> GeneratorExit, it is either one you threw into a generator yourself
>> (it just bounced back, meaning the generator didn't handle it at all),
>> or one that was thrown into you. But the pattern of catching
>> GeneratorExit and responding by returning a value is a reasonable
>> extension of the pattern of catching GeneratorExit and doing other
>> cleanup.
>
> (TLDR version: I'm -1 on Guido's modified close() semantics if there
> is no way to get the result out of a yield from expression that is
> terminated by GeneratorExit, but I'm +1 if we tweak PEP 380 to make
> the result available on the reraised GeneratorExit instance, thus
> allowing framework authors to develop ways to correctly unwind a
> generator stack in response to close())
>
> Stepping back a bit, let's look at the ways a framework may "close" a
> generator-based operation (or substep of a generator).
>
> 1. Send in a sentinel value (often None, but you could easily reuse
> the exception types as sentinel values  as well)
> 2. Throw in GeneratorExit explicitly
> 3. Throw in StopIteration explicitly

Throwing in StopIteration seems more unnatural than any other option.

> 4. Throw in a different specific exception
> 5. Call g.close()
>
> Having close() return a value only helps with the last option, and
> only if the coroutine is set up to work that way. Yield from also
> isn't innately set up to unwind correctly in any of these cases,
> without some form of framework based signalling from the inner
> generator to indicate whether or not the outer generator should
> continue or bail out.

Yeah, there is definitely some kind of convention needed here. A
framework or app can always choose not to use g.close() for this
purpose (heck, several current frameworks use yield to return a value)
and in some cases that's just the right thing. Just like in other flow
control situations you can often choose between sentinel values,
exceptions, or something else (e.g. flag variables that must be
explicitly tested).

> Now, *if* close() were set up to return a value, then that second
> point makes the idea less useful than it appears. To go back to the
> simple summing example (not my
> too-complicated-for-a-mailing-list-discussion version which I'm not
> going to try to rehabilitate):
>
> def gtally():
>  count = tally = 0
>  try:
>    while 1:
>      tally += yield
>      count += 1
>  except GeneratorExit:
>    pass
>  return count, tally

I like this example.

> Fairly straightforward, but one of the promises of PEP 380 is that it
> allows us to factor out some or all of a generator's internal logic
> without affecting the externally visible semantics. So, let's do that:
>
>  def gtally2():
>    return (yield from gtally())

And I find this a good starting point.

> Unless the PEP 380 yield from expansion is changed, Guido's proposed
> "close() returns the value on StopIteration" just broke this
> equivalence for gtally2() - since the yield from expansion turns the
> StopIteration back into a GeneratorExit, the return value of
> gtally2.close is always going to be None instead of the expected
> (count, tally) 2-tuple. Since the value of the internal call to
> close() is thrown away completely, there is absolute nothing the
> author of gtally2() can do to fix it (aside from not using yield from
> at all).

Right, they could do something based on the (imperfect) equivalency
between "yield from f()" and "for x in f(): yield x".

> To me, if Guido's idea is adopted, this outcome is as
> illogical and unacceptable as the following returning None:
>
>  def sum2(seq):
>    return sum(seq)

Maybe.

> We already thrashed out long ago that the yield from handling of
> GeneratorExit needs to work the way it does in order to serve its
> primary purpose of releasing resources, so allowing the inner
> StopIteration to propagate with the exception value attached is not an
> option.
>
> The question is whether or not there is a way to implement the
> return-value-from-close() idiom in a way that *doesn't* completely
> break the equivalence between gtally() and gtally2() above. I think
> there is: store the prospective return-value on the GeneratorExit
> instance and have the yield from expansion provide the most recent
> return value as it unwinds the stack.
>
> To avoid giving false impressions as to which level of the stack
> return values are from, gtally2() would need to be implemented a bit
> differently in order to *also* convert GeneratorExit to StopIteration:
>
>  def gtally2():
>    # The PEP 380 equivalent of a "tail call" if g.close() returns a value
>    try:
>      yield from gtally()
>    except GeneratorExit as ex:
>      return ex.value

Unfortunately this misses the goal of equivalency between gtally() and
your original gtally2() by a mile. Having to add extra except clauses
around each yield-from IMO defeats the purpose.

> Specific proposed additions/modifications to PEP 380:
>
> 1. The new "value" attribute is added to GeneratorExit as well as
> StopIteration and is explicitly read/write

I already posted an argument against this.

> 2. The semantics of the generator close method are modified to be:
>
>  def close(self):
>    try:
>      self.throw(GeneratorExit)
>    except StopIteration as ex:
>      return ex.value
>    except GeneratorExit:
>      return None # Ignore the value, as it didn't come from the
> outermost generator
>    raise RuntimeError("Generator ignored GeneratorExit")
>
> 3.  The GeneratorExit handling semantics for the yield from expansion
> are modified to be:
>
>        except GeneratorExit as _e:
>            try:
>                _m = _i.close
>            except AttributeError:
>                pass
>            else:
>                _e.value = _m() # Store close() result on the exception
>            raise _e
>
> With these modifications, a framework could then quite easily provide
> a context manager to make the idiom a little more readable and hide
> the fact that GeneratorExit is being caught at all:
>
> class GenResult():
>    def __init__(self): self.value = None
>
> @contextmanager
> def generator_return():
>    result = GenResult()
>    try:
>      yield
>    except GeneratorExit as ex:
>      result.value = ex.value
>
> def gtally():
>  # The CM suppresses GeneratorExit, allowing us
>  # to convert it to StopIteration
>  count = tally = 0
>  with generator_return():
>    while 1:
>      tally += yield
>      count += 1
>  return count, tally
>
> def gtally2():
>  # The CM *also* collects the value of any inner
>  # yield from expression, allowing easier tail calls
>  with generator_return() as result:
>    yield from gtally()
>  return result.value

I agree that you've poked a hole in my proposal. If we can change the
expansion of yield-from to restore the equivalency between gtally()
and the simplest gtally2(), thereby restoring the original refactoring
principle, we might be able to save it. Otherwise I declare defeat.
Right now I am too tired to think of such an expansion, but I recall
trying my hand at one a few nights ago and realizing that I'd
introduced another problem. So this does not look too hopeful,
especially since I really don't like extending GeneratorExit for the
purpose.

-- 
--Guido van Rossum (python.org/~guido)