frange() question

[Carsten Haese]

On Thu, 2007-09-20 at 18:55 +0000, John J. Lee wrote:
> Functions are never generators, senso stricto.  There are "generator
> functions", which *return* (or yield) generators when you call them.

Actually, a generator function is a function that returns a generator.
The generator, in turn, is an object that wraps the iterator protocol
around a resumable function. When the generator's next() method is
called, it resumes its wrapped function that runs until it yields or
finishes. When the wrapped function yields a value, it is suspended, and
the yielded value is returned as the result of next(). If the wrapped
function finished, next() raises StopIteration.

Generators are a special case of iterators. Any object that implements
the iterator protocol (which among other less important things mandates
a next() method that returns the next value or raises StopIteration) is
an iterator. Generators simply have their special way (by calling into a
resumable function) of implementing the iterator protocol, but any
stateful object that returns a value or raises StopIteration in next()
is an iterator.

> It's true sometimes people refer to generator functions as simply
> "generators", but in examples like the above, it's useful to remember
> that they are two different things.
> 
> In this case, frange isn't a generator function, because it doesn't
> yield.  Instead, it returns the result of evaluating a generator
> expression (a generator).  The generatator expression plays the same
> role as a generator function -- calling a generator function gives you
> a generator object; evaluating a generator expression gives you a
> generator object.  There's nothing to stop you returning that
> generator object, which makes this function behave just like a regular
> generator function.

Generator expressions still "yield", but you don't see the yield in the
Python source code. Observe:

>>> def gf1():
...   for i in (1,2,3):
...     yield i
... 
>>> def gf2():
...   return (i for i in (1,2,3))
... 
>>> g1 = gf1()
>>> g2 = gf2()
>>> g1
<generator object at 0xb7f66d8c>
>>> g2
<generator object at 0xb7f66f4c>
>>> dir(g1)
['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'close', 'gi_frame', 'gi_running',
'next', 'send', 'throw']
>>> dir(g2)
['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__',
'__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'close', 'gi_frame', 'gi_running',
'next', 'send', 'throw']
>>> import dis
>>> dis.dis(g1.gi_frame.f_code.co_code)
          0 SETUP_LOOP         19 (to 22)
          3 LOAD_CONST          4 (4)
          6 GET_ITER       
    >>    7 FOR_ITER           11 (to 21)
         10 STORE_FAST          0 (0)
         13 LOAD_FAST           0 (0)
         16 YIELD_VALUE    
         17 POP_TOP        
         18 JUMP_ABSOLUTE       7
    >>   21 POP_BLOCK      
    >>   22 LOAD_CONST          0 (0)
         25 RETURN_VALUE   
>>> dis.dis(g2.gi_frame.f_code.co_code)
          0 SETUP_LOOP         18 (to 21)
          3 LOAD_FAST           0 (0)
    >>    6 FOR_ITER           11 (to 20)
          9 STORE_FAST          1 (1)
         12 LOAD_FAST           1 (1)
         15 YIELD_VALUE    
         16 POP_TOP        
         17 JUMP_ABSOLUTE       6
    >>   20 POP_BLOCK      
    >>   21 LOAD_CONST          0 (0)
         24 RETURN_VALUE   

As you can see, except for a minor optimization in the byte code, there
is no discernible difference between the generator that's returned from
a generator expression and the generator that's returned from the
generator function. Note in particular that the byte code for g2
contains a YIELD_VALUE operation just like g1 does.

In fact, generator expressions are merely a short-hand notation for
simple generator functions of a particular form. The generator
expression

g = (expr(x) for x in some_iterable)

produces a generator that is almost indistinguishable from, and
operationally identical to, the generator produced by the following
code:

def __anonymous():
  for x in some_iterable:
    yield expr(x)
g = __anonymous()
del __anonymous

Hope this helps,

-- 
Carsten Haese
http://informixdb.sourceforge.net