Iterators

[Daniel Dittmar]

Chris Lyon wrote:
> coming from a Vb'ish sort of background, issues like iterators and
> list comprehensions appear at first site to be a confusion. However
> since I have a great deal of respect for the large brains in a jar
> (LBIAJ) that post on this list and construct my favourite language I
> realise I'm missing something.

Perhaps it's best explained by the problem these constructs were intended to
solve:

= list comprehension =
There seems to be a lot of code like the following:

# transforming a list
result = []
for element in sourcelist:
    result.append (transform (element))

# filtering a list
result = []
for element in sourcelist:
    if passesFilter (element):
        result.append (element)

# both filtering and transforming
result = []
for element in sourcelist:
    if passesFilter (element):
        result.append (transform (element))

Some thought that this is a lot of typing and possibly obfuscates the intent
of the code, so list comprehension syntax was borrowed from the Haskell
programming language:

result = [transform (element) for element in sourcelist]
result = [element for element in sourcelist if passesFilter (element)]
result = [transform (element) for element in sourcelist if passesFilter
(element)]

In the old days, this was done using map (for transformation) and filter
(for filtering):
result = map (transform, sourcelist)
result = filter (passesFilter, sourcelist)
result = map (transform, filter (passesFilter, sourcelist))

This looks actually shorter than list comprehension. But assume that
transform and passesFilter are member functions. Then it becomes
result = map (lambda e: e.transform (), sourcelist)
result = filter (lambda e: e.passesFilter (), sourcelist)
result = map (lambda e: e.transform (), filter (lambda e: e.passesFilter (),
sourcelist))
versus
result = [element.transform () for element in sourcelist]
result = [element for element in sourcelist if element.passesFilter ()]
result = [element.transform () for element in sourcelist if
element.passesFilter ()]

The main disadvantages of list comprehension:
- you can't insert print statements
- you can't transform first and filter last unless you nest list
comprehension
  result = [outer for outer in [transform (inner) for inner in sourcelist]
if passesFilter (outer)]

2.4 will also introduce 'generator comprehension':
- no [] around the construct
- more efficient because the list isn't built in memory

= Iterators =
The code to read all lines of a file used to be as follows:
while 1:
    line = stream.readline ()
    if not line:
        break # end of file
    pass # do something with line

which is cumbersome to write.

Or
for line in stream.readlines ():
    pass # do something with line

which will consume too much memory if the file is large.

So what was needed was some kind of object:
- that could be used in a for loop
- doesn't put everything in memory

Another example would be potentially large SQL queries, where you want to
loop over all the rows.

A simple implementation to iterate over the lines of a file
class LineIterator:
    def __init__ (self, fname):
        self.stream = open (fname, 'r')

    def next (self):
        line = self.stream.readline ()
        if not line:
            raise StopIteration
        return line

    def __iter__ (self):
        return self

Disadvantage of using iterators:
- you can read them only once
- print doesn't write anything useful

Daniel