[Python-ideas] Generators are iterators

[Chris Barker]

On Fri, Dec 12, 2014 at 5:58 AM, Oscar Benjamin <oscar.j.benjamin at gmail.com>
wrote:

> Okay well this is not clear from the PEP (and bear in mind that I did
> take the time to read through all of the lengthy threads on this
> subject before posting here). I haven't read the most recent changes
> but when I did read the PEP the justification appeared to stem from
> claims that were confusing at best and plain false at worst e.g.
> "generators are not iterators".
>

I don't think there is much technical confusion here, but there seems to
be, so, so that highlights that it's very important to write vary carefully
about all this. I"m going to try this:

1) It's not clear what is meant by an "iterator" -- when we use the term is
seems like we are talking about a type, when we are really talking about
"anything that conforms to the iterator protocol", so I'm going to
introduce a new term: TIIP (Type Implementing the Iterator Protocol) --
maybe that's too cute, but role with it for now.

2) there are a number of TIPPS in the standard library: things like
tupleiterator and friends, the range object, etc.

3) there are essentially two (and a half) ways for python code authors to
write their own, custom TIIP:
  1) write a class that has a __iter__ and __next__ methods that "do the
right thing"
  2a) write a generator function
  2b) write a generator expression

The latter create a generator type, which is a specific TIIP that come
built in to the standard library. Generator functions are a particularly
nifty syntax for creating TIIPs -- i.e. I'm pretty sure that there is
nothing you can do with a generator function that you can't do by writing a
custom TIIP class. But generator functions provide really nifty syntax that
"does a lot of the bookeeping for you"

This is pretty much how I've taught this stuff in the past. But what this
PEP does is change generators a bit, so that they do even more of the book
keeping for you -- that sure seems reasonable enough when phrased that way.

So I agree with Oscar (and others) here the issue of what happens to a
StopIteration that is raised inside a TIIP be another call is an issue that
effects not only generators, but also custom TIIP classes. I think is wold
be nice if this were addressed in both cases the same way, but that has
been proposed and rejected for what I, at least, think are sound technical
reasons.

And the PEP has been accepted so the only thing left to bike-shed about is
how it discussed.

So why it it worth changing the behavior of generators?

1) we can -- custom classes are, well, custom, so are free to do what
author wants them to do. Generators are built-in classes and can therefor
be controlled  by python itself.

2) generator function have a different, cleaner, more compact way to
express intent. while they create generators that conform to the iteration
protocol -- i.e. raise a StopIteration when done, the author does not have
to explicitly do that -- you can write a generator function without any
awareness of StopIteration ( I might argue that you won't get far if you
don't understand all that, but...) So having generators automatically "do
the right thing" when they call another TIIP inside themselves conforms
well to the spirit of generator functions: clean compact code that does the
book keeping for you.

3) when you write a custom TIIP class, you have complete control over what
it does, and have to be thinking about when StopIteration is raise. So you
are more likely to think about how you want to handle a StopIteration that
may be raised by another call to a TIP inside your code. You also want to
be able to control that depending on circumstances, so probably wouldn't
want magic Exception mangling anyway.

Not sure if this helps the PEP writing, but I've got some good notes for
the next time I teach this...

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141212/d4cecc65/attachment.html>