inverse of izip

[Steven Bethard]

Satchidanand Haridas <sharidas at zeomega.com> wrote in message news:<mailman.1941.1092909660.5135.python-list at python.org>...
> Could you expand on what you mean by exhaust the iterators too early?
> 
> The reason I ask is that the * operator is applied to 
> ((1,1),(2,2),....(9,9)). The operation of the 
> itertools.izip(range10(),range10()) is completed before the * operation 
> is applied. And the iter() simply converts the result of the inverse 
> izip operation into an iterator.  I hope the above was not too 
> confusing. :-)

Yeah, the difference is a little subtle here.  What we have before you
use the * operator is an iterator that will yield (1,1) then (2,2) up
to (9,9).  Note that we don't actually have the tuple
((1,1),(2,2),....(9,9)) yet, just an iterator that will produce the
same elements.  If your list is very large and you don't want to keep
it all in memory at once, it's crucial that we have the iterator here,
not the tuple.

When you use the * operator, Python converts the iterable following
the * into the argument list of the function.  This means that if
you're using an iterable, it reads all of the elements of the iterable
into memory at once.  That's why my range10 iterators printed
"exhausted" after the * application -- all their elements had been
read into memory.  Again, if your list is very large, this is a bad
thing because you now have all the elements of the list in memory at
the same time.  My other solution (well, Peter Otten's correction of
my solution) never has the whole list in memory at the same time --
each time enumerate generates a tuple and it's index, each of the
iterators returned by starzip generates their appropriate items.[*]

Steve

[*] Of course, if you exhaust one of the iterators before the others,
itertools.tee's implicit cache will actually store all the elements,
so starzip would really only be efficient if you wanted to iterate
through the sub-iterators in lockstep.  This means you'd probably want
to itertools.izip them back together at some point, but being able to
starzip them means you can wrap the individual iterators with extra
functionality if necessary.