[Python-Dev] Re: Minimal 'stackless' PEP using generators?

[Terry Reedy]

"Clark C. Evans" <cce at clarkevans.com> wrote in message
news:20040823153923.GA50408 at prometheusresearch.com...
> I just read the thread 'Stackless Python' in June 2004 on python-dev and
> was wondering if you'd comment

Sure

> on a simpler cooperative mechanism, via a small hack to generators:

If it is possible, as proposed, maybe at least a medium hack?

> 1.  The PEP would introduce a new 'builtin' class called 'Cooperate'
>
> 2.  Generator semantics would be altered so that 'yield X', where X
>     is an instance of Cooperate, would automagically propigate to
>     the outer-most non-generator.

-1

Skipping over the generic objections of 'hard to teach to beginners' and
'opens to door to a hundred other such specialized features':

1. This alters and complexifies the current generator concept.  At present,
'generator' (the return type of any generator function) is an instance of
the iterator concept, as is 'iterator' (the return type of iter()) and any
class with appropriate .__iter__ and .next methods.  A generator .next
method is a resumable function; yield is like return except not permanent.
The local namespace is not destroyed and instruction pointer not reset for
the next call.

>From the outside, a generator .next method is, I believe, indistinguishable
in Python (excluding possible implementation-specific voodoo) from an
interator-type .next method and behaviorally indistingueshable from any
other iterator-protocol parameterless .next method.  As far as I know, the
difference between a generator and other iterators is ease of writing
(sometimes *much* easier) and speed of execution, not external behavior.

2. This imposes an isinstance(x, magic.Collaborate) function call cost on
every yield in every generator.  (I do not believe the alternative of
imposing the same cost on every function call in every generator will work
since I do not believe that callers can dependably know whether objects are
returned or yielded.  See comments on example.)

3. 'automagic' it is.  The pertinent runtime call stack contains not
generator objects but .next methods (or in CPython, method-wrappers).  So I
presume you are proposing that yielded Collaborates magically plunk
themselves into the first frame up the stack not associated such objects.

4. If the returned to frame does not re-call the passed over suspended
frames, they and anything they hold onto are leaked until gc'ed.

>     >>> def MyCooperate(magic.Cooperate):
>     >>>     __str__(self): return "cooperate"
>     >>>
>     >>> def top():
>     >>>     yield "one"
>     >>>     yield MyCooperate()
>     >>>     yield "two"
>     >>>
>     >>> def middle():
>     >>>     """ intermediate generator _only_ sees one and two """
>     >>>     for x in top():
>     >>>         print "middle", x
>     >>>         yield x

Here, the effect you seek could be accomplished by an explicit 'if not
isinstance(x, Collaborate):' before the print statement (or, in general,
the block of code that uses x).  But what if top is rewritten as
def top(): return ("one", MyCooperate(), "two")
Do you really want the behavior of middle to change, as with your proposal?

>     >>> def lower():
>     >>>     """ this is not a generator, so it sees cooperate """
>     >>>     for x in middle():
>     >>>         print "lower", x

Now add 'if isinstance(x, Cooperate): return' and the suspended middle will
be left in limbo.  Not very tidy ;-)

Terry J. Reedy