itertools.izip brokeness

[Bengt Richter]

On 4 Jan 2006 15:20:43 -0800, "Raymond Hettinger" <python at rcn.com> wrote:

>Paul Rubin wrote:
>> What do you think of my suggestion of passing an optional arg to the
>> StopIteration constructor, that the caller can use to resume the
>> iterator or take other suitable recovery steps?  Maybe this could
>> interact with PEP 343 in some useful way.
>
>Probably unworkable.  Complex to explain and use.   Makes the API
>heavy.  Hard to assure retro-fitting for every possible kind of
>iterator or iterable.
>
>Am not sure of the best solution:
>
>1. Could add an optional arg to zip()/izip() with a mutable container
>to hold a final, incomplete tuple:  final=[]; zip(a,b,leftover=final).
>This approach is kludgy and unlikely to lead to beautiful code, but it
>does at least make accessible data that would otherwise be tossed.
>
>2. Could add a new function with None fill-in -- essentially an
>iterator version of map(None, a,b).  Instead of None, a user specified
>default value would be helpful in cases where the input data stream
>could potentially have None as a valid data element.  The function
>would also need periodic signal checks to make it possible to break-out
>if one the inputs is infinite. How or whether such a function would be
>used can likely be answered by mining real-world code for cases where
>map's None fill-in feature was used.
>
>3. Could point people to the roundrobin() recipe in the
>collections.deque docs -- it solves a closely related problem but is
>not exactly what the OP needed (his use case required knowing which
>iterator gave birth to each datum).
>
>4. Could punt and leave this for straight-forward while-loop coding.
>Though the use case seems like it would be common, there may be a
>reason this hasn't come up since zip() was introduced way back in
>Py2.0.
>
>5. Could create an iterator wrapper that remembers its last accessed
>item and whether StopIteration has been raised.  While less direct than
>a customized zip method, the wrapper may be useful in contexts other
>than zipping -- essentially, anywhere it is inconvenient to have just
>consumed an iterator element.  Testing the wrapper object for
>StopIteration would be akin to else-clauses in a for-loop. OTOH, this
>approach is at odds with the notion of side-effect free functional
>programming and the purported benefits of that programming style.
>
>
6. Could modify izip so that one could write

    from itertools import izip
    zipit = izip(*seqs)	# bind iterator object to preserve access to its state later
    for tup in zipit:
        # do something with tup as now produced
    for tup in zipit.rest(sentinel):
        # tup starts with the tuple that would have been returned if all sequences
        # had been sampled and sentinel substituted where StopIteration happened.
        # continuing until but not including (sentinel,)*len(seqs)

This would seem backwards compatible, and also potentially allow you to use the rest mode
from the start, as in

    for tup in izip(*seqs).rest(sentinel):
        # process tup and notice sentinel for yourself


Regards,
Bengt Richter