[Python-ideas] "Iteration stopping" syntax [Was: Is this PEP-able? for X in ListY while conditionZ:]

[Andrew Barnert]

From: Nick Coghlan <ncoghlan at gmail.com>

Sent: Saturday, June 29, 2013 10:45 PM


> On 30 June 2013 14:53, Andrew Barnert <abarnert at yahoo.com> wrote:
>>  If you don't raise an exception through a listcomp, that cost is 
>> basically running one more opcode and loading a few more bytes into memory. It 
>> adds less than 1% for even a trivial comp that loops 10 times, or for a 
>> realistic but still simple comp that loops 3 times.
> 
> Comprehensions translate to inline loops. The fact that the "stop()"
> hack works for generator expressions is just a quirky result of the
> line between "return" and "raise StopIteration" in a 
> generator
> function being extraordinarily blurry - it has nothing to with
> breaking out of a loop.

I think it's a pretty obvious consequence of the fact that a genexp builds an iterator, and raising StopIteration stops any iterator.


And that's why the stop() hack terminates a nested genexp completely rather than just one loop: because it's the entire genexp that's an iterator, not each individual nested clause.

> You can't get the same semantics for other comprehensions without

> introducing the yield/return distinction, which is astonishingly slow
> by comparison (as you need to suspend and resume the generator frame
> on each iteration).

Sure you can. All you have to do is put a try/except around the outer loop. Taking your example:

    [(x, y) for x in range(10) for y in range(10) if y < 3 or stop()]


This would now map to:

    a = []

    try:
        for x in range(10):
            for y in range(10):
                if y < 3 or stop():
                    a.append((x, y))
    except StopIteration:
        pass

And that change is enough to give it _exactly_ the same semantics as

    list((x, y) for x in range(10) for y in range(10) if y < 3 or stop())

… but without the 40% performance hit from the yields. The only overhead added is a single SETUP_EXCEPT.

But really, despite the origin of the idea, my reason for liking it has little to do with iteration stopping, and much more to do with making the language simpler. The key is that this change would mean that [comp] becomes semantically and behaviorally identical to list(genexp), except for being 40% faster.

Put another way:

I think the language would be better if, instead of documenting comprehensions and generator expressions largely independently and in parallel, we simply said that [comp] has the same effect as list(genexp) except that if you raise StopIteration with a comp it passes through. (The fact that this is true today is by no means obvious, but that's part of the problem I want to solve, not an argument against solving it.)

If we removed that one difference, it would be even simpler. I don't think the difference buys us anything, and the cost of eliminating it is a relatively simple patch with minimal performance impact. (As a minor side benefit, that would also mean you could use the StopIteration hack in comps, but I still don't think we'd want to recommend doing that.)