What is executed when in a generator

[Michael]

[ I've got no idea of your skill level, but since you say you feel a bit
  lost I'll take baby steps here. Apologies if it's too low or high :-)
  I'm also taking baby steps because less people understand generators than
  they might. After all, most new programmers often want something like
  generators when they start programming, but life gets in their way.
]

Jerzy Karczmarczuk wrote:

> Could you tell me please where can I read something in depth about the
> semantics of generators? I feel a bit lost.

Watch what happens if you take your example (I'm not convinced it's the
best example for this, but it is your example :-) on the python shell:

>>> gl=0
>>> def gen(x):
...      global gl
...      gl=x
...      yield x
...
>>> s=gen(1)

Clearly this has done something. What you haven't understood by the looks
of things is what and why. Let's take this session a bit further. Let's
find out the value of s:
>>> s
<generator object at 0x40397a8c>

OK, so the call to the generator function returned a generator. That
should make sense to you. This will hopefully make more sense if we make
a few more calls to 'gen' and look at their values.

>>> t=gen(2)
>>> t
<generator object at 0x40397bec>

As you'd expect the next call *also* creates a generator object. You'll
note that it has a different location ( "at 0x....." ) meaning they're
different objects.

Do that a few more times and we see the same:
>>> u=gen(3)
>>> u
<generator object at 0x403979ec>
>>> v=gen(4)
>>> v
<generator object at 0x40397c0c>

What this is doing is creating a generator object which can then be
asked repeatedly to run, and yield values. 

A more accurate term might be iterating over the generator object. However
the /idea/ is that you create the object wrapping a context of/for running
that can suspend its own control and return intemediate values. Clearly
the generator also needs a way of telling us its finished iterating. Let's
make that more concrete. You call the .next() method of the object, and
when it's done it raises a StopIteration exception.

The way we do this with your code is as follows:

>>> s.next()
1

Yay! That's the "1" that you were expecting to see. In your function you
also updated a global "gl". This is probably not a wise idea, but hey,
let's look at what happened to that value:

>>> gl
1

Again, this is what you should have expected - since it's what you were
after.

Again, looking at your function, "yield x" was the last statement, so you
should be wondering what happens next:

>>> s.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
StopIteration

As you can see we get told that it's dropped off the end. If we try calling
again:

>>> s.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
StopIteration

If we retry with the generator objects created above, we can see:

>>> t=gen(2)
>>> u=gen(3)
>>> v=gen(4)
>>> t.next()
2
>>> gl
2
>>> v.next()
4
>>> gl
4

>>> u.next()
3
>>> gl
3

Hopefully that's a clearer explanation of what's actually going on with
your code. A more complex example is based on the example in the PEP:

def configurable_fib(base1=1, base2=1):
    a, b = base1, base2
    while 1:
        yield a
        a,b = b, a+b

>>> def configurable_fib(base1=1, base2=1):
...     a, b = base1, base2
...     while 1:
...         yield a
...         a,b = b, a+b
>>> normal_fib = configurable_fib(1,1)
>>> normal_fib.next()
1
>>> normal_fib.next()
1
>>> normal_fib.next()
2
>>> normal_fib.next()
3

We can then create another one and ask that for values:

>>> normal_fib2 = configurable_fib(1,1)
>>> normal_fib.next(), normal_fib2.next()
(5, 1)
>>> normal_fib.next(), normal_fib2.next()
(8, 1)
>>> normal_fib.next(), normal_fib2.next()
(13, 2)

Or we can create a few, with unusual bases for the fibonacci sequence and
find their first few values. Let's take the bases of 1 to 10.

>>> fibs = [ configurable_fib(x,x) for x in range(1,11) ]
>>> print [ x.next() for x in fibs ]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> print [ x.next() for x in fibs ]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> print [ x.next() for x in fibs ]
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
>>> print [ x.next() for x in fibs ]
[3, 6, 9, 12, 15, 18, 21, 24, 27, 30]
>>> print [ x.next() for x in fibs ]
[5, 10, 15, 20, 25, 30, 35, 40, 45, 50]

This shows that returning a generator *object* rather than just a value
when you call the generator function is a useful behaviour, even if
initially when learning generators it's slightly counter intuitive.

One way to really understand them though is to try building something
with a whole load of generators. For those purposes, I wrote a tutorial
for our project which is based very much around having lots of generators.
The tutorial is made up of a set of learning exercises:

    * http://kamaelia.sourceforge.net/MiniAxon/

We've tested this tutorial on a couple of novices at work (they learn
python one week and get this tutorial the next), and they've found it
relatively simple. The first hadn't done any programming before, except
a small amount of VB - he was a pre-university trainee. The second was
a university vacation trainee who'd done 2 years, but had no experience
of the ideas in the tutorial or python before joining us.

It's specifically targeted at novices and might be of use when seeing the
possibilities of what you can actually use generators for.

{ Incidentally if anyone reading this, not just yourself, decides to give
  it a go, I'd really appreciate hearing back from you what you thought
  of it, easy/difficult, clear/unclear, what level of experience you have,
  etc. People don't have to really, I'm posting this because I've noticed
  a couple of people have tried this so far as a means to trying to
  understand generators :-) }

Best Regards and hope the above is useful,


Michael.