writable iterators?

[Neal Becker]

Ian Kelly wrote:

> On Wed, Jun 22, 2011 at 3:54 PM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:
>> Fortunately, that's not how it works, and far from being a "limitation",
>> it would be *disastrous* if iterables worked that way. I can't imagine
>> how many bugs would occur from people reassigning to the loop variable,
>> forgetting that it had a side-effect of also reassigning to the iterable.
>> Fortunately, Python is not that badly designed.
> 
> The example syntax is a non-starter, but there's nothing wrong with
> the basic idea.  The STL of C++ uses output iterators and a quick
> Google search doesn't turn up any "harmful"-style rants about those.
> 
> Of course, there are a couple of major differences between C++
> iterators and Python iterators.  FIrst, C++ iterators have an explicit
> dereference step, which keeps the iterator variable separate from the
> value that it accesses and also provides a possible target for
> assignment.  You could say that next(iterator) is the corresponding
> dereference step in Python, but it is not accessible in a for loop and
> it does not provide an assignment target in any case.
> 
> Second, C++ iterators separate out the dereference step from the
> iterator advancement step.  In Python, both next(iterator) and
> generator.send() are expected to advance the iterator, which would be
> problematic for creating an iterator that does both input and output.
> 
> I don't think that output iterators would be a "disaster" in Python,
> but I also don't see a clean way to add them to the existing iterator
> protocol.
> 
>> If you want to change the source iterable, you have to explicitly do so.
>> Whether you can or not depends on the source:
>>
>> * iterators are lazy sequences, and cannot be changed because there's
>> nothing to change (they don't store their values anywhere, but calculate
>> them one by one on demand and then immediately forget that value);
> 
> No, an iterator is an object that allows traversal over a collection
> in a manner independent of the implementation of that collection.  In
> many instances, especially in Python and similar languages, the
> "collection" is abstracted to an operation over another collection, or
> even to the results of a serial computation where there is no actual
> "collection" in memory.
> 
> Iterators are not lazy sequences, because they do not behave like
> sequences.  You can't index them, you can't reiterate them, you can't
> get their length (and before you point out that there are ways of
> doing each of these things -- yes, but none of those ways use
> sequence-like syntax).  For true lazy sequences, consider the concept
> of streams and promises in the functional languages.
> 
> In any case, the desired behavior of an output iterator on a source
> iterator is clear enough to me.  If the source iterator is also an
> output iterator, then it propagates the write to it.  If the source
> iterator is not an output iterator, then it raises a TypeError.
> 
>> * mutable sequences like lists can be changed. The standard idiom for
>> that is to use enumerate:
>>
>> for i, e in enumerate(seq):
>> seq[i] = e + 42
> 
> Unless the underlying collection is a dict, in which case I need to do:
> 
> for k, v in d.items():
>     d[k] = v + 42
> 
> Or a file:
> 
> for line in f:
>     # I'm not even sure whether this actually works.
>     f.seek(-len(line))
>     f.write(line.upper())
> 
> As I said above, iterators are supposed to provide
> implementation-independent traversal over a collection.  For writing,
> enumerate fails in this regard.


While python may not have output iterators, interestingly numpy has just added 
this capability.  It is part of nditer.  So, this may suggest a syntax.

There have been a number of responses to my question that suggest using indexing 
(maybe with enumerate).  Once again, this is not suitable for many data 
structures.  c++ and stl teach that iteration is often far more efficient than 
indexing.  Think of a linked-list.  Even for a dense multi-dim array, index 
calculations are much slower than iteration.

I believe the lack of output iterators is a defienciency in the python iterator 
concept.