[Python-ideas] Change how Generator Expressions handle StopIteration

[Nick Coghlan]

On 3 November 2014 13:01, Guido van Rossum <guido at python.org> wrote:
> On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
>>>
>>> This is related to the fact that, although the docs imply otherwise,
>>> [COMP] isn't exactly equivalent to list(COMP),
>>
>>
>> That purported equivalence is a common meme, which I may have helped
>> spread.
>
>
> I may have started it. I was aware of the non-equivalence (only
> mostly-equivalence) in Python 2 and I wanted to make then identical in
> Python 3 -- having one construct being exactly equivalent to another reduce
> the amount of explaining needed. Unfortunately, people had started to depend
> on the (in my *current* opinion deplorable) behavior of generator
> expressions in the face of StopIteration thrown by arbitrary parts of the
> expression or condition, and the equivalence is still imperfect. At least
> the variable leakage has been fixed.

I think I'm guilty as well - when I was working on the Python 3
changes, getting the *scoping* behaviour to be identical between
comprehensions and generator expressions was one of the key
objectives, so I regularly described it as making "[x for x in seq]"
equivalent to "list(x for x in seq)".

I unfortunately didn't notice the remaining exception handling
differences at the time, or we might have been able to do something
about it for 3.0 :(

> However, I don't think this idea has panned out. I haven't done a survey,
> but I have a feeling that in most cases where an explicit next() call is
> used (as opposed to a for-loop) there's a try/except Stopiteration around
> it, and a fair amount if time is wasted debugging situations where a
> StopIteration unexpectedly escapes and silently interrupts some loop over an
> unrelated generator (instead of loudly bubbling up to the top and causing a
> traceback, which would be more debuggable). And the use case of raising
> StopIteration from a condition used in a generator expression is iffy at
> best (it makes the condition function hard to use in other contexts, and it
> calls to attention the difference between generators and comprehensions).
>
> So I will go out on a limb here and say that this was a mistake and if we
> can think of easing the transitional pain it would be a good thing to fix
> this eventually.

Having had to do the dance to work around the current behaviour in
contextlib, I'm inclined to agree - there's a significant semantic
difference between the "this iterable just terminated" StopIteration,
and the "something threw StopIteration and nobody caught it", and the
current model hides it.

However, I also see potentially significant backwards compatibility
problems when it comes to helper functions that throw StopIteration to
terminate the calling generator - there would likely need to be some
kind of thread local state and a helper "iterexit()" builtin and
"PyIter_Exit()" C API to call instead of raising StopIteration
directly.

Making such a change would involve a lot of code churn just to phase
out a relatively obscure issue that mostly just makes certain bugs
harder to diagnose (as was the case with the generator expression
based izip implementation in this thread), rather than directly
causing bugs in its own right.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia