[Python-ideas] Generators are iterators

[Oscar Benjamin]

On 12 December 2014 at 20:06, Terry Reedy <tjreedy at udel.edu> wrote:
>
>
> On 12/12/2014 7:42 AM, Oscar Benjamin wrote:> On 12 December 2014 at 10:34,
> Nick Coghlan <ncoghlan-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:
>
>>> specific problem deemed worthy of being fixed is that the presence of
>>> "yield" in a function body can implicitly suppress StopIteration
>>> exceptions raised elsewhere in that function body (or in functions it
>>> calls).
>
> This makes no sense to me either.

I'm wondering if Nick is referring to two different things at once: On
the one hand "yield from" which is an iterator-consumer suppresses
StopIteration from a child iterator and on the other hand every
generator interacts with a parent iterator-consumer that will catch
any StopIteration it raises. When generator function f "yield from"s
to generator function g both things occur in tandem in two different
frames.

<snip>
>
>> The problem (insofar as it is) is a problem for all
>> iterators since all iterators interact with iterator-consumers and it
>> is the iterator-consumer that catches the StopIteration. Here is an
>> example using map:
>>
>>>>> def func(x):
>> ...     if x < 0:
>> ...         raise StopIteration
>> ...     return x ** 2
>
> This function violates the implicit guideline, which I think should be more
> explicit in the doc, that the only functions that should expose
> StopIteration to the outside world are iterator.__next__ functions and the
> corresponding builtin next().  It is not surprising that this violation
> causes map to misbehave.  However, changing the .__next__ methods of map and
> filter iterators should be a new thread, which I will write after posting
> this.

The function was just supposed to generate the effect. More plausible
examples can be made. I tripped over this myself once when doing
something with iterators that required me to extract the first item
before processing the rest. A simple demonstration:

def sum(iterable):
    iterator = iter(iterable)
    total = next(iterator)
    for item in iterator:
        total += item
    return total

results = list(map(sum, data_sources))

> The g.f. PEP says that generator functions should also follow this guideline
> and not raise StopIteration (but should always return when not yielding).

> ...
>>> The change in the PEP is to change that side effect such that
>>> those exceptions are converted to RuntimeError rather than silently
>>> suppressed - making generator function bodies behave less like
>>> __next__ method implementations.
>>
>> Fine but generator function bodies are __next__ method
>> implementations.
>
> This is about as wrong, or at least as confusing, as saying that the
> function passed to map is a __next__ method implementation. There is only
> one generator.__next__, just as there is only one map.__next__. Both
> .__next__ methods execute the user function (or function frame) passed in to
> the corresponding .__init__ to calculate the next value to return.  So the
> external functions assist the __next__ methods but are not __next__ methods
> themselves. In particular, they should not raise StopIteration.

The idea is that it happens by accident.


Oscar