access to generator state

[Tim Peters]

[Neal D. Becker]
> ... 
> Only one problem.  Is there any way to access the state of a
> generator externally?  In other words, the generator saves all it's
> local variables.  Can an unrelated object then query the values of
> those variables?  (In this case, to get at intermediate results)

It's the same as asking whether you can peek at the internal state of
any function -- "a generator" in CPython is really just a Python
function stack frame, essentially the same as a non-generator's stack
frame.  The primary difference is that when a function returns, its
frame is decref'ed and typically gets garbage-collected then; when a
generator yields, its frame isn't decref'ed, and the
generator-iterator holds on to the frame for resumption.  In this
sense, Python's generators are quite literally "resumable functions".

Python-the-language doesn't define any way to peek inside one function
from another.  CPython-the-implementation can be exploited, as a
generator-iterator in CPython has a gi_frame attribute referencing the
frame.  You can pick that apart in Python like any other CPython stack
frame.  For example, here's a toy program:

"""
def fib(a, b):
    i = 0
    yield a
    i = 1
    yield b
    while True:
        started_loop = True
        i, a, b = i+1, b, a+b
        yield b
        if b > 12:
            break

f = fib(0, 1)
for val in f:
    print val, f.gi_frame.f_locals
"""

and here's its output:

0 {'i': 0, 'a': 0, 'b': 1}
1 {'i': 1, 'a': 0, 'b': 1}
1 {'i': 2, 'a': 1, 'b': 1, 'started_loop': True}
2 {'i': 3, 'a': 1, 'b': 2, 'started_loop': True}
3 {'i': 4, 'a': 2, 'b': 3, 'started_loop': True}
5 {'i': 5, 'a': 3, 'b': 5, 'started_loop': True}
8 {'i': 6, 'a': 5, 'b': 8, 'started_loop': True}
13 {'i': 7, 'a': 8, 'b': 13, 'started_loop': True}

Note some subtleties:  despite possible appearance, it's *not* the
case that the local vrbl 'started_loop' doesn't exist before it's
assigned to.  Local variables are wholly determined at compile time,
and all exist as soon as a function is entered.  Locals aren't
normally represented internally in a dict, either.  That you see a
dict at all here, and that it suppresses entries for unbound locals,
is all the result of fancy code executed as a side effect of
referencing the "f_locals" attribute of a frame (f_locals is akin to a
Python property with a "getter" function).

Note too that, as when getting a dict via the locals() builtin inside
a function, mutating the dict has no defined effect (it may or may not
have any visible effect, depending on accidents that aren't, and never
will be, documented or guaranteed).