[Python-ideas] Propagating StopIteration value

[Terry Reedy]

On 10/9/2012 11:34 AM, Serhiy Storchaka wrote:
> On 09.10.12 10:51, Greg Ewing wrote:
>> Where we seem to disagree is on
>> whether returning a value with StopIteration is part of the
>> iterator protocol or the generator protocol.

There is a generator class but no 'generator protocol'. Adding the extra 
generator methods to another iterator class will not give its instances 
the suspend/resume behavior of generators. That requires the special 
bytecodes and flags resulting from the presence of 'yield' in the 
generator function whose call produces the generator.

> Is a generator expression work with the iterator protocol or the
> generator protocol?

A generator expression produces a generator, which implements the 
iterator protocol and has the extra generator methods and suspend/resume 
behavior.

Part of the iterator protocol is that .__next__ methods raise 
StopIteration to signal that no more objects will be yielded. A value 
can be attached to StopIteration, but it is irrelevant to it use as a 
'done' signal. Any iterator .__next__ method.  can raise or pass along 
StopIteration(something). Whether 'something' is even seen or not is a 
different question. The main users of iterators, for statements, ignore 
anything extra.

> A generator expression eats a value with StopIteration:
>
>  >>> def G():
> ...     return 42
> ...     yield
> ...
>  >>> next(x for x in G())
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> StopIteration
>
> Is it a bug?

Of course not. A generator expression is an abbreviation for a def 
statement defining a generator function followed by a call to that 
generator function. (x for x in G()) is roughly equivalent to

def __():
   for x in G():
     yield x
   # when execution reaches here, None is returned, as usual

_ = __()
del __
_  # IE, _ is the value of the expression

A for loop stops when it catches (and swallows) a StopIteration 
instance. That instance has served it function as a signal. The for 
mechanism ignores any attributes thereof.

The generator .__next__ method that wraps the generator code object (the 
compiled body of the generator function) raises StopIteration if the 
code object ends by returning None. So the StopIteration printed in the 
traceback above is a different StopIteration instance and come from a 
different callable than the one from G that stopped the for loop in the 
generator. There is no sensible way to connect the two. Note that a 
generator can iterate through multiple iterators, like map and chain do.

If the generator stops by raising StopIteration instead of returning 
None, *that* StopIteration instance is passed along by the .__next__ 
wrapper. (This may be an implementation detail, but it is currently true.)

 >>> def g2():
	SI = StopIteration('g2')
	print(SI, id(SI))
	raise SI
	yield 1

 >>> try: next(g2())
except StopIteration as SI:
	print(SI, id(SI))
	
g2 52759816
g2 52759816

If you want any iterator to raise or propagate a value-laden 
StopIteration, you must do it explicitly or avoid swallowing one.

 >>> def G():  return 42;     yield

 >>> def g3(): # replacement for your generator expression
	it = iter(G())
	while True:
		yield next(it)

 >>> next(g3())
Traceback (most recent call last):
   File "<pyshell#29>", line 1, in <module>
     next(g3())
   File "<pyshell#28>", line 4, in g3
     yield next(it)
StopIteration: 42  # now you see the value

Since filter takes a single iterable, it can be written like g3 and not 
catch the StopIteration of the corresponding iterator.

def filter(pred, iterable):
   it = iter(iterable)
   while True:
     item = next(it)
     if pred(item):
       yield item
   # never reaches here, never returns None

Map takes multiple iterables. In 2.x, map extended short iterables with 
None to match the longest. So it had to swallow StopIteration until it 
had collected one for each iterator. In 3.x, map stops at the first 
StopIteration, so it probably could be rewritten to not catch it. 
Whether it makes sense to do that is another question.

-- 
Terry Jan Reedy