Generator (re-)definition within a loop

[Terry Reedy]

On 6/21/2010 3:29 AM, Pierre Reinbold wrote:

> On 06/18/2010 11:48 PM, Terry Reedy wrote:

>> Let's apply Reedy's Rule: when you have trouble understanding a function
>> expression, replace it with the (near) equivalent def statement. (Among
>> other advantages, one can insert print calls!)
>>
>> Genexps, like lambdas, are specialized function expressions.
>>
>> def augment(s_of_s, s):
>>       for x in s_of_s:
>>            for y in s:
>>                 yield x+[y]
>>
>> def gen_product(*args):
>>      pools = map(tuple, args)
>>      result = [[]]
>>      for pool in pools:
>>          result = augment(result,pool)
>>      for prod in result:
>>          yield tuple(prod)
>>
>> print(list(gen_product("ABC", "xy")))
>>
>>>>> #3.1
>> [('A', 'x'), ('A', 'y'), ('B', 'x'), ('B', 'y'), ('C', 'x'), ('C', 'y')]
>
> Very instructive post ! Thank you !

> Just trying to understand, I have apply the Reedy's Rule in an attempt
> to reproduce the same behaviour as the generator expression.

I intentionally said '(near) equivalent' because, as you seem to have 
found, exactly reproducing a bug in a function expression with a def 
statement may be difficult to impossible. The form I applied above was 
'equivalent to what one *thought* one was writing'. If one just wants to 
fix the bug without completely understanding it, that should be enough.

> The idea, I
> guess, is to (re-)define the generator function inside the loop.

To produce a similar bug, in this case, yes.
>
> My first try gives this:
>
> def badgen_product1(*args, **kwds):
>      pools = map(tuple, args)
>      result = [[]]
>      for pool in pools:
>          def result():
>              for x in result():
>                  for y in pool:
>                      yield x+[y]
>      for prod in result():
>          yield tuple(prod)
>
> But this does not reproduce the generator expression, it leads naturally
> to an infinite recursion (which is what I expected first for the
> generator expression btw)
>
>      for x in result():
> RuntimeError: maximum recursion depth exceeded
>
> Another try to avoid infinite recursion:
>
> def badgen_product2(*args, **kwds):
>      pools = map(tuple, args)
>      result = [[]]
>      for pool in pools:
>          def augments():
>              for x in result:
>                  for y in pool:
>                      yield x+[y]
>          result = augments()
>      for prod in result:
>          yield tuple(prod)
>
> And this one gives the infamous:
>
>      for x in result:
> ValueError: generator already executing
>
> Which seems to indicate that the lazy evaluation leads to eventually
> bind everything to the same generator.
>
> So, neither of my attempts reproduce the behaviour of the generator
> expression. What do I miss here ?

See above. Exact reproduction is not always possible.
>
> Thank for your help,

You are welcome. I hope you learned something. Wrestling with code is a 
good way to do that. I love Python because it makes it so easy to do that.

-- 
Terry Jan Reedy