[Python-ideas] PEP 479: Change StopIteration handling inside generators

[Wolfgang Maier]

On 11/25/2014 12:03 AM, Chris Angelico wrote:
> On Tue, Nov 25, 2014 at 9:53 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>> In addition, the PEP leaves an iterator's __next__() method as the only
>> reasonable place where user-code should raise StopIteration.
>> So I would like to argue that instead of just turning StopIteration into
>> some other error when it's about to bubble out of a generator frame, it
>> should be converted whenever it bubbles out of *anything except an
>> iterator's __next__()*. This would include comprehensions, but also any
>> other code.
>
> There'd have to be a special case for next(), where StopIteration is
> part of the definition of the function. The question then becomes,
> what's the boundary where StopIteration is converted?
>
> The current proposal is quite simple. All the conversion happens in
> the one function that (re)starts a generator frame, gen_send_ex() in
> Objects/genobject.c. To do this for other functions, there'd need to
> be some way of saying which ones are allowed to raise StopIteration
> and which aren't.
>

Well, I'm not familiar with every implementation detail of the 
interpreter so I can't judge how difficult to implement certain things 
would be, but one solution that I could think of is:

allow StopIteration to be raised anywhere, but let it bubble up only 
*one* frame.
So if the next outer frame does not deal with it, the exception would be 
converted to UnhandledStopIteration (or something else) when it's about 
to bubble out of that outer frame.
The builtin next() would simply reset the frame count by catching and 
reraising StopIteration raised inside its argument (whether that's an 
iterator's __next__ or a generator; note that in this scenario using 
raise StopIteration instead of return inside a generator would remain 
possible).

Examples of what would happen:

using next on a generator that raises StopIteration explicitly:
=> next catches the error and reraises StopIteration

using next on a generator that returns:
=> next behaves like currently, raising StopIteration

using next on the __next__ method of an iterator:
=> next catches the error and reraises StopIteration

every direct call of an iterator's __next__ method:
=> has to be guarded by a try/except StopIteration

Likewise in the first three cases, the calling frame, which resumes when 
next returns, (and only this frame) is given a chance to handle the 
error. If that doesn't happen (i.e. the error would bubble out) it gets 
converted.

So different from the current PEP where a StopIteration must be dealt 
with explicitly using try/except only inside generators, but bubbles up 
everywhere else, here StopIteration will be special everywhere, i.e., it 
must be passed upwards explicitly through all frames or will get converted.

Back to Steven's generator expression vs comprehension example:

iterable = [iter([])]
list(next(x) for x in iterable)

would raise UnhandledStopIteration since there is no way, inside the 
generator expression to catch the StopIteration raised by next(x).

... and if that's all complete nonsense because of some technical detail 
I'm not aware of, then please excuse my ignorance.

Wolfgang