[Python-Dev] Re: Reiterability

[Phillip J. Eby]

At 10:17 AM 10/18/03 -0700, Guido van Rossum wrote:

>[Phillip]
> > I think I phrased my question poorly.  What I should have said was:
> >
> > "Should iterator expressions preserve the reiterability of the base
> > expression?"
>
>(An iterator expression being something like
>
>   (f(x) for x in S)
>
>right?)

Yes.


> > In essence, this would be done by having an itercomp expression
> > resolve to an object whose __iter__ method calls the underlying
> > generator, returning a generator-iterator.  Thus, any iteration over
> > the itercomp is equivalent to calling a no-arguments generator.  The
> > result is reiterable if the base iterable is reiterable, otherwise
> > not.
>
>OK, I think I understand what you're after.  The code for an iterator
>expression has to create a generator function behind the scenes, and
>call it.  For example:
>
>   A = (f(x) for x in S)
>
>could be translated into:
>
>   def gen(seq):
>       for x in seq:
>           yield f(x)
>   A = gen(S)
>
>(Note that S could be an arbitrary expression and should be evaluated
>only once.  This translation does that correctly.)

Interesting.  That wasn't the semantics I envisioned.  I was thinking 
(implicitly, anyway) that an iterator comprehension was a closure.  That 
is, that S would be evaluated each time.  However, if S is a sequence, you 
don't need to reevaluate it, and if S is another iterator expression that 
preserves reiterability, you still don't need to.  So, in that sense 
there's never a need to


>This allows one to iterate once over A (a generator function doesn't
>allow reiteration).  What you are asking looks like it could be done
>like this (never mind the local names):

Yes, that's actually what I said, but I guess I was once again unclear.


>   def gen(seq):
>       for x in seq:
>           yield f(x)
>   class Helper:
>       def __init__(seq):
>           self.seq = seq
>       def __iter__(self):
>           return gen(self.seq)
>   A = Helper(S)
>
>Then every time you use iter(A) gen() will be called with the saved
>value of S as argument.

Yes, except of course Helper would be a builtin type.


>I don't mind that so much, but I don't think all the extra machinery
>is worth it; the compiler generally can't tell if it is needed so it
>has to produce the reiterable code every time.

It has to produce the generator every time, anyway, presumably as a nested 
function with access to the current locals.  The only question is whether 
it can be invoked more than once, and whether you create the helper 
object.  But maybe that's what you mean, and now you're being unclear 
instead of me.  ;)


>   If you *want* to
>have an iterable instead of an iterator, it's usually easy enough do
>(especially given knowledge about the type of S).

I just tend to wish that I didn't have to think about whether iterators are 
reiterable or not, as it forces me to expose to callers of a function 
whether the value they pass must be an iterator or an iterable.  But I 
don't want to reopen the entire reiterability discussion, as I don't have 
any better solutions and the previously proposed solutions make my head 
hurt just trying to make sure I understand the implications.