[Python-ideas] Generators are iterators

[Oscar Benjamin]

On 12 December 2014 at 13:14, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 12 December 2014 at 22:42, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>> On 12 December 2014 at 10:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> The
>>> specific problem deemed worthy of being fixed is that the presence of
>>> "yield" in a function body can implicitly suppress StopIteration
>>> exceptions raised elsewhere in that function body (or in functions it
>>> calls).
>>
>> The yield causes the function to become a generator function. The
>> frame for a generator function (like for any other function) will
>> allow uncaught exceptions to propagate to the frame above. The
>> difference between generator functions and other functions is that the
>> code in the body of a generator function is executed when someone
>> calls the generator's __next__ method. Since the caller (the iterator
>> consumer) is expecting StopIteration it is treated as the signalling
>> the end of iteration. The yield suppresses nothing; it is the iterator
>> consumer e.g. the for-loop or the list() function etc. which catches
>> the StopIteration and treats it as termination.
>
> Oscar, this is where you're *just plain wrong* from a technical perspective.

Which of the above sentences is technically incorrect?

> The conversion to a generator function also introduces an additional
> layer of indirection at iteration time through the
> generator-iterator's __next__ method implementation.
>
> That indirection layer is the one that converts an ordinary return
> from the generator frame into a StopIteration exception. Historically,
> it also allowed StopIteration from within the frame to propagate out,
> make it impossible to tell from outside the generator iterator whether
> the termination was due to the generator returning normally, or due to
> StopIteraton being raised.

I understand all of this.

> The introduction of "yield from" in Python 3.3, and its use in the
> coroutine design for the asyncio library and similar contexts *makes
> that distinction more important than it used to be*. It even managed
> to introduce obscure bugs around the use of generators to write
> context managers, where factoring out part of the context manager to a
> subgenerator may implicitly suppress a StopIteration exception raised
> in the body of the with statement.
>
> The problem you think PEP 479 is trying to solve *is* the one where
> the discussion started, but it is *not* the one where it ended when
> Guido accepted PEP 479. The problems PEP 479 solves are generator
> specific - they have nothing to do with the iterator protocol in
> general.

Okay well this is not clear from the PEP (and bear in mind that I did
take the time to read through all of the lengthy threads on this
subject before posting here). I haven't read the most recent changes
but when I did read the PEP the justification appeared to stem from
claims that were confusing at best and plain false at worst e.g.
"generators are not iterators".


Oscar