[Python-ideas] Possible PEP 380 tweak

[Jacob Holm]

On 2010-10-26 12:36, Nick Coghlan wrote:
> On Tue, Oct 26, 2010 at 1:14 PM, Guido van Rossum <guido at python.org> wrote:
>> Well, *throwing* it is close()'s job. And *catching* it ought to be
>> pretty rare. Maybe this idiom would be better:
>>
>> def sum():
>>  total = 0
>>  try:
>>    while True:
>>      value = yield
>>      total += value
>>  finally:
>>    return total
> 
> Rereading my previous post that Jacob linked, I'm still a little
> uncomfortable with the idea of people deliberately catching
> GeneratorExit to turn it into a normal value return to be reported by
> close(). That said, I'm even less comfortable with the idea of
> encouraging the moral equivalent of a bare except clause :)

What Nick said. :)


> I see two realistic options here:
> 
> 1. Use GeneratorExit for this, have g.close() return a value and I
> (and others that agree with me) just get the heck over it.
> 

This has the benefit of not needing an extra method/function and an
extra exception for this style of programming.  It still has the
refactoring problem I mention below.  That might be fixable in a similar
way though.  (Hmm thinking about this gives me a strong sense of deja-vu).


> 2. Add a new GeneratorReturn exception and a new g.finish() method
> that follows the same basic algorithm Guido suggested, only with a
> different exception type:
> 
> class GeneratorReturn(Exception): # Note: ordinary exception, unlike
> GeneratorExit
>   pass
> 
> def finish(gen):
>  try:
>    gen.throw(GeneratorReturn)
>    raise RuntimeError("Generator ignored GeneratorReturn")
>  except StopIteration as err:
>    if err.args:
>      return err.args[0]
>  except GeneratorReturn:
>    pass
>  return None
> 

I like this.  Having a separate function lets you explicitly request a
return value and making it fail loudly when called on an exhausted
generator feels just right given the prohibition against saving the
"True" return value anywhere.  Also, using a different exception lets
the generator distinguish between the "close" and "finish" cases, and
making it an ordinary exception makes it clear that it is *intended* to
be caught.  All good stuff.

I am not sure that returning None when finish() cathes GeneratorReturn
is a good idea though.  If you call finish on a generator you expect it
to do something about it and return a value.  If the GeneratorReturn
escapes, it is a sign that the generator was not written to expect this
and so it likely an error.  OTOH, I am not sure it always is so maybe
allowing it is OK.  I just don't know.

How does it fit with the current PEP 380, and esp. the refactoring
principle?   It seems like we need to special-case the GeneratorReturn
exception somehow.  Perhaps like this:

[...]
  try:
      _s = yield _y
+ except GeneratorReturn as _e:
+     try:
+         _m = _i.finish
+     except AttributeError:
+         raise _e  # XXX RuntimeError?
+     raise YieldFromFinished(_m())
  except GeneratorExit as _e:
[...]

Where YieldFromFinished inherits from GeneratorReturn, and has a 'value'
attribute like the new StopIteration.

Without something like this a function that is written to work with
"finish" is unlikely to be refactorable.   With this, the trivial case
of perfect delegation can be written as:

def outer():
    try:
        return yield from inner()
    except YieldFromFinished as e:
        return e.value

and a slightly more complex case...

def outer2():
    try:
        a = yield from innerA()
    except YieldFromFinished as e:
        return e.value
    try:
        b = yield from innerB()
    except YieldFromFinished as e:
        return a+e.value
    return a+b

the "outer2" example shows why the special casing is needed.  If
outer2.finish() is called while outer2 is suspended in innerA, a
GeneratorReturn would be thrown directly into innerA.  Since innerA is
supposed to be expecting this, it returns a value immediately which
would then be the return value of the yield-from.  outer2 would then
erroneously continue to the "b = yield from innerB()" line, which unless
innerB immediately raised StopIteration would yield a value causing the
outer2.finish() to raise a RuntimeError...

We can avoid the extra YieldFromFinished exception if we let the new
GeneratorReturn exception grow a value attribute instead and use it for
both purposes.  But then the distinction between a GeneratorReturn that
is thrown in by "finish" (which has no associated value) and the
GeneratorReturn raised by the yield-from (which has) gets blurred a bit.

Another idea is to actually replace YieldFromFinished with StopIteration
or a GeneratorReturn inheriting from StopIteration.  That would mean we
could drop the first try-except block in each of the above example
generators because the "finished" result from the inner function is
returned directly anyway.  On the other hand, that could easily lead to
subtle bugs if you forget a try...except block that is actually needed,
like the second block in outer2.


A different way to handle this would be to change the PEP 380 expansion
as follows:

[...]
- except GeneratorExit as _e:
+ except (GeneratorReturn, GeneratorExit) as _e:
[...]

What this means is that only the outermost generator would see the
GeneratorReturn.  If the outermost generator is suspended using
yield-from, and finish() is called.  The inner generator is simply
closed and the GeneratorReturn re-raised.  This version is only really
useful for delegating to generators that *don't* return a value, but it
is simpler and at least it allows *some* use of yield-from with "finish".


> (Why "finish" as the suggested name for the method? I'd prefer
> "return", but that's a keyword and "return_" is somewhat ugly. Pairing
> GeneratorReturn with finish() is my second choice, for the "OK, time
> to wrap things up and complete your assigned task" connotations, as
> compared to the "drop everything and clean up the mess" connotations
> of GeneratorExit and close())

I like the names.  GeneratorFinish might work as well for the exception,
but I like GeneratorReturn better for its connection with "return".


> 
> I'd personally be +1 on option 2 (since it addresses the immediate use
> case while maintaining appropriate separation of concerns between
> guaranteed resource cleanup and graceful completion of coroutines) and
> -0 on option 1 (unsurprising, given my previously stated objections to
> failing to maintain appropriate separation of concerns).
> 

I agree the "finish" idea looks far better for generators without
yield-from.  It is unfortunate that extending it to work with yield-from
isn't prettier that it is though.



> (I should note that this differs from the previous suggestion of a
> GeneratorReturn exception in the context of PEP 380. Those suggestions
> were to use it as a replacement for StopIteration when a generator
> contained a return statement. The suggestion here is to instead use it
> as a replacement for GeneratorExit in order to request
> prompt-but-graceful completion of a generator rather than just bailing
> out immediately).

I agree the name fits this use better than the original.  Too bad some
of my suggestions above are starting to blur the line between
GeneratorReturn and StopIteration again.

- Jacob