Iterator / Iteratable confusion

[Terry Reedy]

"Francis Girard" <francis.girard at free.fr> wrote in message 
news:200502142131.53265.francis.girard at free.fr...

(Note for oldtimer nitpickers: except where relevant, I intentionally 
ignore the old and now mostly obsolete pseudo-__getitem__-based iteration 
protocol here and in other posts.)

Le dimanche 13 Février 2005 23:58, Terry Reedy a écrit :
>> Iterators are a subgroup of iterables. Being able to say iter(it) 
>> without
>> having to worry about whether 'it' is just an iterable or already an
>> iterator is one of the nice features of the new iteration design.

>I have difficulties to represent an iterator as a subspecie of an 
>iteratable

You are not the only one.  That is why I say it in plain English.

You are perhaps thinking of 'iterable' as a collection of things.  But in 
Python, an 'iterable' is a broader and more abstract concept: anything with 
an __iter__ method that returns an iterator.

To make iterators a separate, disjoint species then requires that they not 
have an __iter__ method.  Some problems:
A. This would mean either
    1) We could not iterate with iterators, such as generators, which are 
*not* derived from iterables, or, less severely
    2) We would, usually, have to 'protect'  iter() calls with either 
hasattr(it, '__iter__') or try: iter(it)...except: pass with probably no 
net average time savings.
B. This would prohibit self-reiterable objects, which require .__iter__ to 
(re)set the iteration/cursor variable(s) used by .next().
C. There are compatibility issues not just just with classes using the old 
iteration protocol but also with classes with .next methods that do *not* 
raise StopIteration.  The presence of .__iter__ cleanly marks an object as 
one following the new iterable/iterator protocol.  Another language might 
accomplish the same flagging with inheritance from a base object, but that 
is not Python.

> [snip]...C++ STL where there is a clear (I resist to
> say "clean") distinction between iteratable and iterator.

leaves out self-iterating iterables -- collection objects with a .next 
method.  I am sure that this is a general, standard OO idiom and not a 
Python-specific construct.  Perhaps, ignoring these, you would prefer the 
following nomenclature:
iterob   = object with .__iter__
iterable= iterob without .next
iterator = iterob with .next

Does STL allow/have iterators that are *not* tied to an iterable?

>One of the result of not distinguishing them is that, at some point in 
>your
>programming, you are not sure anymore if you have an iterator or an
>iteratable ; and you might very well end up calling "iter()" or
> "__iter__()" everywhere.

If you iterate with a while loop just after creating a new iterable or 
iterator, then you probably do know which it is and can make the iter() 
call only if needed.  If you while-iterate with a function argument, then 
iter() is a simpler way to be generic than the alternatives in A2 above.

>I am not concerned with the small performance issue involved here

Good.  I think there are small.  The number and time for iterations is far 
more important.

> (as I very seldom am) but with clarity. After all, why should you have to 
>  > call __iter__ on an iterator you just constructed

As I said above, you don't, and most people wouldn't.  The function 
implementing for loops does because *it*, unlike you, only sees the object 
passed and not the code that created the object!

>I have a strong feeling that the problem arises from the difficulty to
> marry the familiar ""for ... in ..."" construct with iterators. [snip]

What difficulty?  For loops accept an iterable and iterate with the derived 
iterator.

Would you really prohibit the use of for loops with generators and other 
non-iterable-derived iterators?  See A1 above.

>To have iterators act as iteratables might very well had been a 
> >compromise to solve the problem.

I think it elegant.  See below.

>I am not sure at all that this is a "nice feature" to consider an iterator 
>at
>the same level that an iteratable.

I think you are too stuck in the STL model.

> It makes it a bit more akward to have the
>"mind impulse", so to speak, to build iterators on top of other iterators 
>to > slightly modify the way iteration is done.

On the contrary, what could be more elegant than
def itermodifier(it):
  for i in it: # where it is often an iterator
     yield modification-of-i
See the itertools module and docs and the examples of chaining iterators.

Terry J. Reedy