[Python-Dev] Termination of two-arg iter()

[Tim Peters]

[Barry]
> The problem that Jeff Epler brought up (extending the list after
> StopIterator was returned, and having a subsequent .next() not give
> StopIterator) has a precedence in dict iterators:
>
> -------------------- snip snip --------------------
> >>> d = {1:2, 3:4}
> >>> it = iter(d)
> >>> for x in d: print x
> ...
> 1
> 3
> >>> d[5] = 6
> >>> it.next()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> RuntimeError: dictionary changed size during iteration
> -------------------- snip snip --------------------
>
> So why doesn't that last .next() also return StopIterator? <wink>.

According to the PEP as it exists, it should.

> StopIterator is a sink state for dict iterators if I don't change the
> size of the dict.

That's an illusion <wink>.  See below for why.

> Shouldn't list and dict iterators should behave similarly for
> mutation (or at least resizing) between .next() calls?

Within a single for-loop, list iterators are constrained to be compatible
with their previous implementation via the __getitem__ protocol.  So, for
example, this must work:

>>> x = [1]
>>> for y in x:
...     print y
...     x.append(y+1)
...     if y == 5:
...         break
...
1
2
3
4
5
>>>

because that's the way "for elt in list" has always worked.  Nothing about
that violates what the PEP says, though (in particular, StopIteration isn't
an issue there, as it's never raised).  It's too difficult to do something
similar for dict iterators, and that's why they raise an exception if they
detect a size change.  However, they *really* want to raise an exception if
the dict mutates, but that's also too hard to do -- checking for a size
change is a cheap & easy but vulnerable approximation.  Ponder the output
from this on a run, and then across several runs:

from random import random

for limit in range(1, 100):
    d = {}
    for i in range(limit):
        d[random()] = 1

    i = d.iterkeys()
    x = list(i)  # exhausts the iterator
    d.popitem()
    d[random()] = 1  # probably mutated, but # of elements is the same
    try:
        print i.next()  # tries poking the iterator again
        print limit, list(i)
    except StopIteration:
        pass

You'll find that this *usually* raises StopIteration on the lone i.next()
call (and you don't get output in those cases).  However, for *some* list
sizes, it's quite likely that poking the iterator again not only produces
another value, but that it can produce several more values.  There's no
predicting how many, which or when, though.

It's a bit of a stretch to call that "a feature" too <wink>.