[Python-ideas] Generators are iterators

[Steven D'Aprano]

In an effort to try to keep this already too-long thread a bit 
shorter, I'm going to trim drastically.

On Sat, Dec 13, 2014 at 09:43:02PM -0800, Andrew Barnert wrote:
> On Dec 13, 2014, at 17:17, Steven D'Aprano <steve at pearwood.info> wrote:

> > I read your reply to Chris last night, and it didn't make sense to me 
> > then and it still doesn't make sense to me now :-( Hence my request for 
> > a concrete example.
> 
> You didn't ask for a concrete example, you provided the exact same 
> answer as Chris but with different names.

My first post in this sub-thread ended with:

"So I'm not sure which mysterious object you are referring to that
doesn't have a standard name. Can you give a concrete example?"

But moving along:


> > I am still unclear as to what this mystery unnamed thing is. It's not a 
> > generator function, because that is called a generator function. It's 
> > not the thing you get when you call a generator function, because that 
> > is called a generator or an iterator, depending on context.
> 
> Again, it's the thing used by the generator instance's __next__ 
> method: the frame and code objects.

Aha, now the penny drops. Given a generator object, is has attributes 
holding a frame object and a code object (which it shares with the 
generator function that created it):

py> def gen(): yield 1
...
py> it = gen()
py> it.gi_frame
<frame object at 0xb7b8f95c>
py> it.gi_code
<code object gen at 0xb7b6e660, file "<stdin>", line 1>
py> it.gi_code is gen.__code__
True

Both have standard names: "frame object" and "code object" respectively.


> I don't know how else to explain it. Look at the members stored by the 
> generator type: a frame and a code object.

That's exactly the concrete example I was looking for.


> So, going back to the original point: Generator instances are not 
> different from iterators (except that it's a subtype, of course). The 
> builtin generator type is likewise no different than a custom iterator 
> class. The generator body is a normal function body. The only thing 
> that isn't like an iterator is the frame. Which is not an important 
> distinction.

Well, there are a few other differences as well. `generator` is a 
concrete type, `Iterator` is an abstract type, and `iterator` can refer 
to either a specific iterator type:

py> class X:
...     def __getitem__(self, i): pass
...
py> iter(X())
<iterator object at 0xb7af4b0c>

or speaking more generically, to anything which obeys the iterator 
protocol. More importantly, generators have abilities that other 
iterators don't have: you can send data into a generator, not just get 
data out of them.


> I think Chris was making a different distinction, claiming that 
> generators are not an Iterator type (which they aren't, but only 
> because they're _instances_ of an Iterator type, the builtin type 
> generator), 

I'm not sure what Chris was trying to say, but I've seen various other 
people maintain that generators aren't iterators. As you say, that is 
wrong:

py> from collections.abc import Iterator
py> isinstance(gen(), Iterator)
True

The *class* `generator` is an Iterator (sub-)type, but not an Iterator 
itself:

py> issubclass(type(gen()), Iterator)
True
py> isinstance(type(gen()), Iterator)
False

which is no different from instances of int being numbers, but int 
itself (the class) not being a number.


Thank you for helping me understand what you were trying to say.



-- 
Steven