[issue10109] itertools.product with infinite iterator cause MemoryError.

[Robert Xiao]

Robert Xiao added the comment:

I think this _is_ a bug. Most of the itertools are quite thrifty with memory, and exceptions are explicitly documented.

The itertools generally only require memory proportional to the number of iterations consumed. However, `itertools.product` requires an enormous up-front memory cost even if only a few iterations are performed. There are good reasons to extract only a few iterations from a `product` - for example, a search procedure where only the first item in the product is infinite:

    for year, day in product(count(2010), range(365)):
        if rare_event(year, day):
            break

Or, in an application I recently tried, to count occurrences of something up to a limit:

    prod = product(count(), range(n))
    filtered = ifilter(pred, prod)
    want = list(islice(filtered, 101))
    if len(want) > 100:
        print "Too many matches"

Having `product` greedily consume all of its input iterators is rather antithetical to the notion of itertools being lazy generators, and it breaks efficient composability with the other itertools.

The fix should be fairly straightforward: like tee, maintain lists of items already generated as the input iterators are exhausted, and add a note that "product"'s memory usage may slowly grow over time (as opposed to growing to maximum size immediately).

----------
nosy: +nneonneo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10109>
_______________________________________