[Tutor] iterators generators

[Gonçalo Rodrigues]

----- Original Message -----
From: "Gerardo Arnaez" <garnaez@yahoo.com>
To: <tutor@python.org>
Sent: Tuesday, April 01, 2003 4:10 PM
Subject: [Tutor] iterators generators


> Hi all, I've been learning a lot,
> really enjoyed list comprehesions.
> but cant seem to get my head around iterators and
> generators, anyone know of any good reading, Ive read
> the PEP's and just didnt grok it
> I also read the IBM Charming python and again failed
> to grok it
>

I'll stand up and try to explain iterators and generators. First a
definition: An object <whatever> is iterable (or is an iterable) if



iter(<whatever>)



doesn't raise an exception (TypeError to be more exact). Iterables are nice
because they allow for loops, e.g.



for elem in <iterable>:

    <whatever>



Examples: list, tuple, file, etc.

>>> print iter([])

<iterator object at 0x01126098>

>>>



Now iter() is a builtin function and when called on an iterable it returns
an *iterator* which may or may not be the same thing as the iterable itself.



>>> a = []

>>> print id(a), id(iter(a))

17990976 17999536



As you can see the list iterable is different from the iterator itself. I
think (but I'm not sure) that files are scheduled in 2.3 to become their own
iterators. Anyway, what the iterator object is supposed to encapsulate is
the traversal through the elements of the iterable. It does this by having a
method, next(), that gives the next element (whatever that may be) or raises
a StopIteration exception if there is nothing left to spew.



When Python encounters a



for elem in <iterable>:

    <whatever>



it basically is the same: get the iterator of iterable and sucessively bind
elem to whatever the iterator's next method returns until a StopIteration
exception is raised.



Note that an iterator is also an iterable. But the iterator of an iterator
is the iterator itself (usually - I suppose there is some more exotic code
out there where this rule is broken).



>>> b = iter(a)

>>> print b

<iterator object at 0x01129C68>

>>> print b is iter(b)

1



If you want to code your own iterable just define __iter__. If it is an
iterator make __iter__ return self (unless you know what you are doing) and
provide a next method.



Now for generators. Generators are a cross-breeding between a function and
an iterable. It is better an example first:



>>> def test(n = 0):

...         start = n

...         while True:

...                     yield start

...                     start += 1

...

>>> print test

<function test at 0x01144A88>



This is a generator because of the presence of the yield keyword in its
body. But, as you can see test *is* a function. When called for the first
time it gives



>>> a = test()

>>> print a

<generator object at 0x01148DA0>



A generator. Now a generator *is* an iterator. To see that:



>>> a.next

<method-wrapper object at 0x010D0FD0>

>>> print id(a), id(iter(a))

18124192 18124192



But it is a special kind of iterator. It remembers all the state about the
"function". Let us go back to our little example to see how it works.



When you first call a.next() (either explicitely or implicitely in a for
loop) it executes the code in the body until the first yield. When it gets
there it "returns" whatever expression is in front of the yield but instead
of discarding the function it "suspends" it. This means two things (at
least): The state of all the local variables is kept *and* the place where
you yielded is also kept. On the next next() call you execute the body of
test with your local variables in the state they were in *and* right from
the place where you last yielded: In our little examlpe, execution resumes
with



start += 1



which bumps start to 1. Execution continues until the next yield.



Generators are mostly used in coding iterators, but in D. Mertz's Charming
Python column you can see other examples for them including exotic
microthreading, state machines, etc.



And that is basically all there is to it.



Hope it helps, with my best regards,

G. Rodrigues