Re-iterable generator comprehensions

[Oren Tirosh]

On Mon, Apr 1, 2002 at 09:53:15AM -2600, J. Random Newbie wrote:
> I am using Python 2.3b1 and I am trying to use generator comprehensions 
> to generate all combinations of a letter and a digit without creating any 
> intermediate lists:
> 
> >>>from __future__ import generator_comprehensions
> >>>letters = [yield chr(i) for i in xrange(chr('a'),chr('z')+1)]
> >>>digits = [yield str(i) for i in xrange(10)]
> >>>letdig = [yield l+d for l in letters for d in digits]
>
> The generator letdig yields 'a0' through 'a9' and stops.  If I remove 
> the yield keywords and use lists it works fine. What's happening here?

According to the current proposal for generator comprehensions the letters
and digits objects are iterators. They can only be traversed once.  If you 
call iter(digits) it just returns itself.  When the letdig generator 
reaches the end of the digits source it calls iter(digits) again, expecting 
a fresh iterator for the same container.  Instead it just gets the same 
exhausted iterator and stops.

The solution is simple: generator comprehensions should not return an 
iterator - they should return an iterable object. This will make their 
behavior much more like real containers and make the code above work 
as expected.

According to the current proposal for generator comprehensions the 
letters object is equivalent to this code:

  def tmpgen():
    for i in xrange(chr('a'),chr('z')+1)]: yield chr(i)
  letters = tmpgen()

With re-iterable generators it would be something like this:

  def tmpgen():
    for i in xrange(chr('a'),chr('z')+1)]: yield chr(i)
  letters = ReGenerator(tmpgen)

where the class ReGenerator is defined as:

  class ReGenerator:
    def __init__(self, genfunc):
      self.__iter__ = genfunc

This object more closely emulates a list or other container: each time
iter(letters) is called it returns a fresh, independent iterator.  Unless
the generator or any of its sources are affected by some global variables 
it should produce the same stream of objects again and again.

The should also apply to the functions xmap, xfilter and xzip.  They 
should return an iterable object, not an iterator.  The xrange built-in 
function already works this way: it returns an iterable xrange object 
emulating a container.  Each time iter(<xrange object>) is called it 
returns a fresh, independent iterator.

   from-__future__-ly yours,

	Oren Tirosh