[Python-ideas] Change how Generator Expressions handle StopIteration

[Nick Coghlan]

On 3 November 2014 15:59, Guido van Rossum <guido at python.org> wrote:
> On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> However, I also see potentially significant backwards compatibility
>> problems when it comes to helper functions that throw StopIteration to
>> terminate the calling generator - there would likely need to be some
>> kind of thread local state and a helper "iterexit()" builtin and
>> "PyIter_Exit()" C API to call instead of raising StopIteration
>> directly.
>
>
> That's what I was afraid of. Can you point me to an example of code that
> depends on this that isn't trivial like Andrew Barnert's ensure_positive()
> example? I think that particular example, and the category it represents,
> are excessive cleverness that abuse the feature under discussion -- but you
> sound like you have helpers for context managers that couldn't be easily
> dismissed like that.

Sorry, I didn't mean to give that impression. I'm in a similar
situation to you in that regard - any specific examples I can think of
trip my "that's too obscure to be maintainable" alarm (if there is a
reasonable use case, I'd expect it to involve full generators, rather
than generator expressions).

The code in contextlib relies on the way *generators* handle
StopIteration, and if understand your proposal correctly, that would
remain unchanged - only ordinary function calls would convert
StopIteration to a different exception type, preserving the functional
equivalence of a generator returning, raising StopIteration, or having
StopIteration thrown into a yield point (it's that last one that
contextlib relies on).

>> Making such a change would involve a lot of code churn just to phase
>> out a relatively obscure issue that mostly just makes certain bugs
>> harder to diagnose (as was the case with the generator expression
>> based izip implementation in this thread), rather than directly
>> causing bugs in its own right.
>
>
> Maybe. But the real-life version of that bug can be *really* hard to find,
> and that's usually the kind of thing we'd like to fix.
>
> FWIW the implementation of my proposal is easy to describe (which the Zen
> approves of): when a StopIteration leaves a frame, replace it with some
> other exception (either a new, custom one or maybe just RuntimeError),
> chaining the original StopIteration.

That's far more elegant than the stateful possibilities I was considering.

So generators would continue to leave StopIteration untouched
(preserving the equivalence between returning from the frame and
explicitly raising StopIteration from the generator body), and only
ordinary function invocations would gain the StopIteration ->
UnexpectedStopIteration translation (assuming we went with a new
exception type)?

> It's the consequences that are hard to survey and describe in this case (as
> they affect subtle code depending on the current semantics).

Aye. I'm reasonably OK with the notion of breaking overly clever (and
hence hard to follow) generator expressions, but I'm a little more
nervous about any change that means that factoring out "raise
StopIteration" in a full generator function would stop working.

That said, such a change would bring generator functions fully into
line with the "flow control should be locally visible" principle that
guided both the with statement design and asyncio - only a local
return or raise statement could gracefully terminate the generator,
StopIteration from a nested function call would always escape as the
new exception type. If you wanted to factor out a helper function that
terminated the generator you'd have to do "return yield from helper()"
rather than just "helper()".

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia