[Python-ideas] Change how Generator Expressions handle StopIteration

[Terry Reedy]

On 11/1/2014 12:50 PM, Guido van Rossum wrote:
> I think you're on to something. But I think both your examples have a
> problem, even though your second one "works".

Both versions are buggy in that iizip() yields () infinitely, while 
zip() yields nothing.  Fixes below.

> If we weren't forced by backward compatibility I would have made it much
> harder for StopIteration to "leak out". Currently a generator can either
> return or raise StopIteration to signal it is done, but I think it would
> have been better if StopIteration was treated as some kind of error in
> this case.

This would require some sort of additional special casing of 
StopIteration that we do not have now.  Currently, it is limited to 
'for' loops expecting and catching StopIteration as a signal to stop 
iterating.  That is rather easy to understand.

 > Basically I think any time a StopIteration isn't caught by a
> for-loop or an explicit try/except StopIteraton, I feel there is a bug
> in the program,

Outside of generator functions (and expressions), I agree as I cannot 
think of an exception when it is not.  This has come up on Python list.

 > or at least it is hard to debug.

Code within generator functions is different.  Writing "raise 
StopIteration" instead of "return" is mostly a waste of keystrokes.  As 
for next(it), StopIteration should usually propagate, as with an 
explicit raise and not be caught.  The code below that 'works' (when it 
does work), works because the StopIteration from next(it) (when there is 
at least one) propagates to the list comp, which lets it pass to the 
generator, which lets it pass to the generator user.

> I'm afraid that ship has sailed, though...
>
> On Sat, Nov 1, 2014 at 7:56 AM, yotam vaknin
> <tomirendo at gmail.com
> <mailto:tomirendo at gmail.com>> wrote:

>     I would like to purpose that generator expressions will not catch
>     StopIteration exception, if this exception did not come from the
>     iterated object's __next__ function specifically.

For the purpose of your example, all instances of StopIteration are the 
same and might as well be the same instance.

Since to my understanding generators and g.e.s already do not catch the 
StopIterations you say you want not caught, and since you need for it to 
not be caught in the code below, I do not understand exactly what you 
propose.

 >     So generator
>     expressions will be able to raise StopIteration by calculating the
>     current value of the Generator.

I cannot understand this.

>     def izip(*args):
>          iters = [iter(obj) for obj in args]
>          while True:
>              yield tuple(next(it) for it in iters)
>
>     a = izip([1,2],[3,4])
>     print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) ()
>     list(izip([1,2],[3,4])) #Currently never returns

Better test code that avoid infinite looping:

a = izip([1,2],[3,4])
for i in range(3):
     print(next(a))

One the third loop, the above prints (), while the below prints a 
traceback.  With a = izip(), both print () 3 times.

The problem is that when next(it) raises, you want the StopIteration 
instance propagated (not immediately caught), so that the 
generator-using code knows that the generator is exhausted.  But the 
tuple call catches it first, so that, in combination with 'while True', 
the user never sees StopIteration  A partial solution is to provoke 
StopIteration before calling tuple, so that it does propagate.  That is 
what the list comp below does.  But if args is empty, so is iters, and 
there is no next(it) to ever raise.  For a complete solution that 
imitates zip and does not require an otherwise useless temporary list, 
replace the loop with this:

     while True:
         t = tuple(next(it) for it in iters)
         if not t:
            return
         yield t

>     Even thought this is the PEP described behaviour, I think this is an
>     unwanted behaviour.

Not if you think carefully about what you want to happen when next(it) 
raises.  I think generators and generators expressions should be left 
alone.

>     I think Generator Expressions should work like List Comprehension in
>     that sense:
>
>     def iizip(*args):
>          iters = [iter(obj) for obj in args]
>          while True:
>              yield tuple([next(it) for it in iters])

This could be fixed with 'if not iters: return' as the second line.

Replacing [genexp] with list(genexp) does not work because the latter, 
unlike the former, catches StopIteration.  This is proof that the two 
are not exactly equivalent, and the such behavior difference I know of 
(excluding introspection, such as with trace).

-- 
Terry Jan Reedy