[Python-ideas] Change how Generator Expressions handle StopIteration

[Andrew Barnert]

On Nov 2, 2014, at 21:59, Guido van Rossum <guido at python.org> wrote:

> On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 3 November 2014 13:01, Guido van Rossum <guido at python.org> wrote:
>> > On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> >>
>> >> On 11/2/2014 2:50 PM, Andrew Barnert wrote:
>> >>>
>> >>> This is related to the fact that, although the docs imply otherwise,
>> >>> [COMP] isn't exactly equivalent to list(COMP),
>> >>
>> >>
>> >> That purported equivalence is a common meme, which I may have helped
>> >> spread.
>> >
>> >
>> > I may have started it. I was aware of the non-equivalence (only
>> > mostly-equivalence) in Python 2 and I wanted to make then identical in
>> > Python 3 -- having one construct being exactly equivalent to another reduce
>> > the amount of explaining needed. Unfortunately, people had started to depend
>> > on the (in my *current* opinion deplorable) behavior of generator
>> > expressions in the face of StopIteration thrown by arbitrary parts of the
>> > expression or condition, and the equivalence is still imperfect. At least
>> > the variable leakage has been fixed.
>> 
>> I think I'm guilty as well - when I was working on the Python 3
>> changes, getting the *scoping* behaviour to be identical between
>> comprehensions and generator expressions was one of the key
>> objectives, so I regularly described it as making "[x for x in seq]"
>> equivalent to "list(x for x in seq)".
>> 
>> I unfortunately didn't notice the remaining exception handling
>> differences at the time, or we might have been able to do something
>> about it for 3.0 :(
>> 
>> > However, I don't think this idea has panned out. I haven't done a survey,
>> > but I have a feeling that in most cases where an explicit next() call is
>> > used (as opposed to a for-loop) there's a try/except Stopiteration around
>> > it, and a fair amount if time is wasted debugging situations where a
>> > StopIteration unexpectedly escapes and silently interrupts some loop over an
>> > unrelated generator (instead of loudly bubbling up to the top and causing a
>> > traceback, which would be more debuggable). And the use case of raising
>> > StopIteration from a condition used in a generator expression is iffy at
>> > best (it makes the condition function hard to use in other contexts, and it
>> > calls to attention the difference between generators and comprehensions).
>> >
>> > So I will go out on a limb here and say that this was a mistake and if we
>> > can think of easing the transitional pain it would be a good thing to fix
>> > this eventually.
>> 
>> Having had to do the dance to work around the current behaviour in
>> contextlib, I'm inclined to agree - there's a significant semantic
>> difference between the "this iterable just terminated" StopIteration,
>> and the "something threw StopIteration and nobody caught it", and the
>> current model hides it.
>> 
>> However, I also see potentially significant backwards compatibility
>> problems when it comes to helper functions that throw StopIteration to
>> terminate the calling generator - there would likely need to be some
>> kind of thread local state and a helper "iterexit()" builtin and
>> "PyIter_Exit()" C API to call instead of raising StopIteration
>> directly.
> 
> That's what I was afraid of. Can you point me to an example of code that depends on this that isn't trivial like Andrew Barnert's ensure_positive() example? I think that particular example, and the category it represents, are excessive cleverness that abuse the feature under discussion

The category is usually represented by the even more trivial and even more abusive example:

    (prime if prime<100 else throw(StopIteration) for prime in primes)

I do see these, mostly on places like StackOverflow, where someone was shown this "cool trick" by someone else, used it without understanding it, and now has no idea how to debug his code. (However, a few people on this list suggested it as an alternative to adding some kind of "syntactic takewhile" to the language, so it's possible not everyone sees it as abuse, even though I think you and others called it abuse back then as well.)

Anyway, I agree that explicitly disallowing it would make the language simpler, eliminate more bugs than useful idioms, and possibly open the door to other improvements. But if you can't justify this abuse as being actually illegal by a reading of the docs in 3.0-3.4, and people are potentially using it in real code, wouldn't that require a period of deprecation before it can be broken?

> -- but you sound like you have helpers for context managers that couldn't be easily dismissed like that.
>  
>> Making such a change would involve a lot of code churn just to phase
>> out a relatively obscure issue that mostly just makes certain bugs
>> harder to diagnose (as was the case with the generator expression
>> based izip implementation in this thread), rather than directly
>> causing bugs in its own right.
> 
> Maybe. But the real-life version of that bug can be *really* hard to find, and that's usually the kind of thing we'd like to fix.
> 
> FWIW the implementation of my proposal is easy to describe (which the Zen approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration.
> 
> It's the consequences that are hard to survey and describe in this case (as they affect subtle code depending on the current semantics).
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141103/5d4624b0/attachment.html>