need help on generator... (re)

[Peter Otten]

Joh wrote:

>> def gen(iterable, start, end):
>>it = iter(iterable)
>>while True:
>>it, a = tee(it)
>>a = tuple(islice(a, end-1))
>>for sz in xrange(start, len(a)+1):
>>yield a[:sz]
>>it.next()
>>
>> if __name__ == "__main__":
>>print list(gen(range(1, 5), 2, 4))
> 
> please, this one looks interesting, could you explain a bit how it
> works and why it "remain memory-efficient" ?

If you have a generator of the form

def gen1(iterable):
    for i in iterable:
        yield f(i)
    for i in iterable:
        yield g(i)

and then use it

for item in gen1(range(huge_number)):
    do_something_useful_with(item)

it will only work when you feed it with something you can iterate over
multiple times, e. g. a list, not a generator that reads data on every call
of the next() method. That means you must store the data for (to keep it
simple) the lifetime of gen1(). If you can modify the generator to

def gen2(iterable):
    for i in iterable:
        yield f(i)
        yield g(i)

for item in gen2(xrange(huge_number)): # switched from range() to xrange()
    do_something_useful_with(item)

there is no such restriction to the iterable. All data can be read,
processed and garbage-collected immediately.

The gen() generator is a bit more complex in that it has to store a few
adjacent items instead of only one and allows for an arbitrary number of
functions (inlined as yield of an n-tuple) instead of always two functions,
but the idea is the same.

Peter