[Python-ideas] PEP 479: Change StopIteration handling inside generators

[Wolfgang Maier]

On 15.11.2014 10:29, Chris Angelico wrote:
> PEP: 479
> Title: Change StopIteration handling inside generators
> Version: $Revision$
> Last-Modified: $Date$
> Author: Chris Angelico <rosuav at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 15-Nov-2014
> Python-Version: 3.5
> Post-History: 15-Nov-2014
>
>
> Abstract
> ========
>
> This PEP proposes a semantic change to ``StopIteration`` when raised
> inside a generator, unifying the behaviour of list comprehensions and
> generator expressions somewhat.
>
>
> Rationale
> =========
>
> The interaction of generators and ``StopIteration`` is currently
> somewhat surprising, and can conceal obscure bugs.  An unexpected
> exception should not result in subtly altered behaviour, but should
> cause a noisy and easily-debugged traceback.  Currently,
> ``StopIteration`` can be absorbed by the generator construct.
>
>
> Proposal
> ========
>
> If a ``StopIteration`` is about to bubble out of a generator frame, it
> is replaced with some other exception (maybe ``RuntimeError``, maybe a
> new custom ``Exception`` subclass, but *not* deriving from
> ``StopIteration``) which causes the ``next()`` call (which invoked the
> generator) to fail, passing that exception out.  From then on it's
> just like any old exception. [3]_
>

Now that this PEP is going to be accepted, I'm not sure how much sense 
it still makes to suggest an amendment to it, but anyway:

As stated in the abstract one of the goals of the PEP is to unify 
further the behaviour of comprehensions and generator expressions.

With the PEP in place the following example (taken from Steven 
d'Aprano's post on python-list):

iterable = [iter([])]
list(next(x) for x in iterable)

would raise an error just like

[next(x) for x in iterable]

already does today.

However the comprehension currently raises StopIteration, while the 
proposed error for the generator expression would be of a different 
class (supposedly RuntimeError) - so comprehensions and generator 
expressions would still behave a bit (though much less) differently.

In addition, the PEP leaves an iterator's __next__() method as the only 
reasonable place where user-code should raise StopIteration.
So I would like to argue that instead of just turning StopIteration into 
some other error when it's about to bubble out of a generator frame, it 
should be converted whenever it bubbles out of *anything except an 
iterator's __next__()*. This would include comprehensions, but also any 
other code.

(On the side, I guess the current form of the PEP does address 
hard-to-debug bugs caused by nested generators, but what about nested 
__next__ in iterators ? Shouldn't it using the same logic also be an 
error if a next call inside a __next__ method raises an uncaught 
StopIteration ?)

I think such general behavior would make it much clearer that 
StopIteration is considered special and reserved for the iterator 
protocol. Of course, it might mean more broken code if people use 
StopIteration or a subclass for error signaling outside 
generator/iterators, but this PEP will mean backwards incompatibility 
anyway so why not go all the way and do it consistently.

I'm not sure I'd like the pretty general RuntimeError for this (even 
though Guido favors it for simplicity), instead one could call it 
UnhandledStopIteration ?
I imagine that a dedicated class would help in porting, for example, 
python2 code to python3 (which this PEP does not really simplify 
otherwise) since people/scripts could watch out for something specific ?

Thoughts?
Wolfgang