PEP 288: Generator Attributes

[Bengt Richter]

I saw this in the python-dev summary:

===================================
`PEP 288:  Generator Attributes`__
===================================
__ http://mail.python.org/pipermail/python-dev/2002-November/030321.html

Raymond Hettinger has revised `PEP 288`_ with a new proposal on how to
pass things into a generator that has already started.  He has asked
for comments on the changes, so let him know what you think.

.. _PEP 288: http://www.python.org/peps/pep-0288.html

so ok, here's what I think ;-)

I like the general *idea* of __self__. I.e., I think
it could be useful for every object to be able to refer to
itself efficiently without having to do it via a global
name bound to itself.

But this particular use of __self__ seems to me suboptimal.
I gather the purpose is to be able to pass data to a generator
in a way that substitutes for not being able to do it through
the .next() method call. But you can already do something so
close to the suggested use that IMO a change is not worth it
for just that.

I.e., the example in PEP 288 is
_______________________________________________________

        def mygen():
               while True:
                   print __self__.data
                   yield None

           g = mygen()
           g.data = 1
           g.next()                # prints 1
           g.data = 2
           g.next()                # prints 2
_______________________________________________________

and by just passing a mutable argument, you can write the
generator code body *exactly* the same, and write the
calling code practically the same:

 >>> from __future__ import generators
 >>> def mygen(__self__):
 ...     while True:
 ...         print __self__.data
 ...         yield None
 ...
 >>> class NS: pass
 ...
 >>> d = NS()
 >>> g = mygen(d)
 >>> d.data = 1
 >>> g.next()
 1
 >>> d.data = 2
 >>> g.next()
 2

I'd rather see generators extended in a backwards compatible way to allow you to
write the minimal example thus:

        def mygen(data):
               while True:
                   print data
                   yield None

        g = mygen.iter()       # mygen.__class__.iter() gets called with the function instance
        g(1)                   # prints 1 (arg name data is rebound to 1, then g.next() effect is done)
        g(2)                   # prints 2 (similarly)


but also allowing something with varying arguments, e.g.,

    def mygen(*args, **kwds):
        ...

    g = mygen.iter()
    first_result  = g(first_args, blah)
    second_result = g(other_args, foo, bar)
    third_result  = g(a_kw_arg=123)
    fourth_result = g() # also ok


I.e., in addition to having a .next() method for backwards compatibility,
the generator object would have a def __call__(self, *args, **kwargs) method,
to make the generator look like a callable proxy for the function.

When the generator's __call__ method was called, it would rebind the associated
function's arg locals each time, as if calling with arg unpacking like mygen(*args, **kwargs)
-- but then resuming like .next()

Possibly g() could be treated as another spelling of g.next(), skipping arg name rebinding.
A default argument should be allowable, and def mygen(x, y=123): ... would have
x and y rebound as you would expect on the first call, at least. You could argue
whether to use the original default for subsequent calls or allow a rebinding
of a default arg name to act like a default for a subsequent call, but I think
optional args should remain optional for all calls.

Note that a generator could now be also used with map, reduce, and filter as functions
as well as sequence sources. E.g., IWT you could write compile as a string filter.

For backwards compatibility. mygen.iter() would have to set a flag or whatever so
that the first g() call would not call mygen and get a generator, but would just
bind the mygen args and "resume" like .next() from the very start instead.

Calls to g.next() could be intermixed and would just bypass the rebinding of the
mygen arg names.

Another possibility you might get from using mygen.iter() rather than a strange (ISTM)
initial function call to create a generator is that you wouldn't be dependent
on yield-detecting lookahead creating a function with builtin weirdness.

I.e., maybe .iter() could be generalized as a factory method of the function class/type,
which might mean that you could set up to capture the frame of an ordinary function
and allow yields from nested calls -- which would open more multitasking possibilities
using generators.

I.e., yield would look down the stack until it found the first call of a __call__ of
a generator, and apparently return from that, while the generator kept the whole
stack state from the yield down to itself for continuing when called again.

Also, it seems like this would make a generator *relate to and control* an associated
normal function instead of *being* a weirded-up function. And yield would be decoupled
enough that you could probably write exec 'yield' and have it work from a dynamically
determined place, so long as there was the frame from a call to a generator's __call__
somewhere on the stack.

Sorry about handwave content in the above, but the possibilities seem tantalizing ;-)
[posted to c.l.p]

Regards,
Bengt Richter