[Python-Dev] return from a generator [was:PEP 380 (yield from a subgenerator) comments]

[Nick Coghlan]

Carl Johnson wrote:
> I think part of the appeal of using "return" is that return is what's
>  used in ordinary functions, but if you think about it, you already 
> have to make your cooperative multitasking mini-thread different from
>  an ordinary function anyway by sprinkling yields throughout it. If 
> you're going that far, is it really so bad to also change the
> "return" to a "raise ReturnValue"?

The big advantage of "return" over "raise" is that from the point of
view of the current execution frame, "return" is a *non-exceptional* exit.

This means that:
  1. "except" clauses do not execute, but neither do "else" clauses
  2. "with" statements pass the appropriate arguments to __exit__
methods to  indicate that the frame exit is non-exceptional

If you try to use "raise" instead of "return":
  1. All except clauses that match the raised exception will execute
  2. "with" statements will pass the raised exception into their
__exit__ methods

Consider a database access subgenerator that writes sent values to the
database inside a transaction and then returns the number of rows
processed (a *really* rough example just to illustrate the point):

  def process_rows(db, update_cmd):
    rows_processed = 0
    with db.transaction() as cursor:
      while 1:
        values = yield
        if not values:
          return rows_processed
        cursor.run(update_cmd, values)
        rows_processed += 1

That works just fine using return - the changes will be committed
correctly to the database, since this is a non-exceptional exit from the
transaction's point of view. If the result must be returned by raising
an exception instead, then you not only have to change the return
statement into a raise statement, but you have to make sure that it is
moved outside the with statement in order to avoid incorrectly rolling
back the database transaction.

That said, there a couple of fairly straightforward mechanisms I can
think of that allow return statements to be used naturally in
generators, while still picking up cases that are almost certainly errors.

If we don't want to modify existing features (e.g. for loops or
contextlib.contextmanager), then we could add a *peer* exception to
StopIteration called GeneratorReturn. Existing code which only catches
StopIteration would allow the new exception to escape, and the name of
that exception could then easily be Googled or looked up in the
documentation.

The only slight oddity is that a bare return in a generator would still
trigger StopIteration, while a "return None" would probably trigger
GeneratorReturn(None).

Another alternative that avoids the need for a new exception type is to
just modify the for loop handling code in the eval loop to check for a
non-None StopIteration.value and raise RuntimeError if that occurs.
(e.g. RuntimeError("Value set on StopIteration: iterator attempted to
return %r to for loop")). contextlib.contextmanager could easily do
something similar (it already raises RuntimeError whenever the
underlying generator doesn't behave as expected).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------