[Python-ideas] Cofunctions - A New Protocol

[Matt Joiner]

I don't think new keywords should be necessary. A module should be sufficient.
Also why CoExit when you have GeneratorExit? Might as well make it
CoroutineExit.

On Tue, Nov 1, 2011 at 9:24 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> A Coroutine Protocol
> ====================
>
> Here are some thoughts on the design of a new protocol to support
> lightweight
> threads using a mechanism similar to, but distinct from, generators and
> yield-from. Separating the two protocols will make it much easier to support
> suspendable generators, something that is not possible using the cofunction
> mechanism as currently specified in PEP 3152.
>
> The protocol to be described is similar in many ways to the generator
> protocol, and in what follows, analogies will be drawn between the two
> protocols
> where it may aid understanding.
>
>
> API
> ---
>
> This section describes the outward appearance of the coroutine mechanism to
> the programmer.
>
> A coroutine is created using the following constructor:
>
> ::
>
>    coroutine(f, *args, **kwds)
>
> where ``f`` is an object obeying the "coroutine protocol" to be described
> below. Syntactic support will be provided for creating such an object using
> a special form of Python function definition, analogous to a generator.
>
> The result is a "coroutine object" having the following methods:
>
> ``resume(value = None)``
>
>    Resumes execution of the coroutine at the point where it was last
>    suspended. The value, if any, is passed into the coroutine and
>    becomes the return value of the operation that caused the suspension.
>    The coroutine executes until its next suspension point, at which
>    time the ``resume`` call returns with the value passed into the
>    suspension operation.
>
>    (Note: This is analogous to calling next() or send() on a
> generator-iterator.
>    Suspension of a coroutine is analogous to a generator executing a
>    ``yield`` operation.)
>
>    If the coroutine has been freshly created, the passed-in value is
>    ignored and the coroutine executes up to its first suspension point.
>
>    If the top level of the coroutine finishes execution without
>    encountering any further suspension points, a ``CoReturn`` exception
>    is raised. This exception has a ``value`` attribute containing the
>    return value from the coroutine.
>
>    (Note: ``CoReturn`` is analogous to the ``StopIteration`` exception
>    raised by an exhausted iterator or generator.)
>
> ``throw(exception)``
>
>    Causes the given exception to be raised in the coroutine at its
>    current suspension point.
>
> ``close()``
>
>    Requests that the coroutine shut down and clean itself up. This is
>    achieved by throwing in a ``CoExit`` exception (analogous to
> ``GeneratorExit``).
>
> It is expected that programmers will not write code that deals directly with
> coroutine objects very often; rather, some kind of driver or scheduler will
> be
> used that takes care of making ``resume()`` calls and handling ``CoReturn``
> exceptions.
>
>
> Cofunctions
> -----------
>
> There will be a special form of Python function called a "cofunction",
> defined
> using the new keyword ``codef`` in place of ``def``. A cofunction provides a
> convenient way of creating an object obeying the coroutine protocol. (This
> is
> similar to how a generator provides a convenient way of creating an object
> obeying the iterator protocol).
>
> Suspension of a cofunction is achieved using the expression
>
> ::
>
>    ``coyield`` [value]
>
> This is analogous to a ``yield`` expression in a generator, and like
> ``yield``,
> it can both provide and receive a value. However, unlike ``yield``, it is
> *not*
> restricted to communicating with the immediate caller. It communicates
> directly
> with the ``resume`` method of the coroutine, however deep the nesting of
> calls
> is between the ``resume`` call and the ``coyield``.
>
> There are some restrictions, however:
>
> * A ``coyield`` is only allowed in the body of a cofunction (a function
> defined
> with ``codef``), not in any other context.
>
> * A cofunction can only be called from the body of another cofunction, not
> in
> any other context.
>
> Exceptions are raised if any of these restrictions are violated.
>
> As a consequence, there must be an unbroken chain of cofunctions (or other
> objects
> obeying the cofunction protocol, see below) making up the call stack from
> the
> ``resume`` method down to the suspension point. A cofunction may call an
> ordinary
> function, but that function or anything called by it will not be able to
> suspend
> the coroutine.
>
> Note that the class of "ordinary functions" includes most functions and
> methods
> written in C. However, it is possible for an object implemented in C to
> participate
> in a coroutine stack by implementing the coroutine protocol below
> explicitly.
>
>
> Coroutine Protocol
> ------------------
>
> As well as the coroutine object, the coroutine protocol involves three other
> kinds
> of objects, "cocallable objects", "coframe objects" and "coiterator
> objects".
>
> A cocallable object has the following method:
>
> ``__cocall__(*args, **kwds)``
>
>    Initiates a suspendable computation. Returns a coframe object.
>
>    (This is analogous to the __iter__ method of an iterable object.)
>
>    May return NotImplemented to signal that the object does not support the
>    coroutine protocol. This enables wrapper objects such as bound methods to
>    reflect whether or not the wrapped object supports the coroutine
> protocol.
>
> A coframe object has the following methods:
>
> ``__resume__(costack, value)``
>
>    There are two purposes for which this method is called: to continue
>    execution from a suspension point, and to pass in the return value
> resulting
>    from a nested call to another cocallable object.
>
>    In both cases, the ``resume`` method is expected to continue execution
> until
>    the next suspension point, and return the value produced by it. If the
>    computation finishes before reaching another suspension point,
>    ``CoReturn(retval)`` must be raised, where ``retval`` is the return value
> of
>    the computation.
>
>    (This method is analogous to the __send__ method of a generator-iterator.
>    With a value of None, it is analogous to the __next__ method of an
> iterator.)
>
>    The currently-executing coroutine object is passed in as the ``costack``
>    parameter. The ``__resume__`` method can make a nested call to another
> cocallable
>    object ``sub`` by performing:
>
>        ``return costack.call(sub, *args, **kwds)``
>
>    No further calls to this coframe will be made until ``obj`` finishes.
> When
>    it does, the ``__resume__`` method of this coframe  is called with the
>    return value from ``sub``.
>
>    It is the responsibility of the coframe object to keep track of whether
> the
>    previous call to its ``__resume__`` method resulted in a suspension or a
> nested
>    call, and make use of the ``value`` parameter accordingly.
>
> ``__throw__(costack, exception)``
>
>    Called to throw an exception into the computation. The coframe may choose
> to
>    absorb the exception and continue executing, in which case ``__throw__``
> should
>    return the value produced by the next exception point or raise
> ``CoReturn`` as
>    for ``__resume__``. Alternatively it may allow the same or a different
> exception
>    to propagate out.
>
>    Implementation of this method is optional. If it is not present, the
> behaviour
>    is as if a trivial ``__throw__`` method were present that simply
> re-raises the
>    exception.
>
> A coiterator is an iterator that permits iteration to be carried out in a
> suspendable
> manner. A coiterator object has the following method:
>
> ``__conext__()``
>
>    Returns a coframe for computing the next item from the iteration. This is
> the
>    coroutine equivalent of an iterator's ``__next__`` method, and behaves
> accordingly:
>    its ``__resume__`` method must return an item by raising
> ``CoReturn(item)``. To
>    finish the iteration, it raises ``StopIteration`` as usual.
>
> To support coiteration, whenever a "next" operation is invoked by a
> cofunction
> (whether implicitly by means of a for-loop or explicitly by calling
> ``next()``)
> a ``__conext__`` method is first looked for, and if found, the operation is
> carried out suspendably. Otherwise a normal call is made to the ``__next__``
> method.
>
>
> Formal Semantics
> ----------------
>
> The semantics of the coroutine object are defined by the following Python
> implementation.
>
> ::
>
>    class coroutine(object):
>
>        #  Public methods
>
>        def __init__(self, main, *args, **kwds):
>            self._stack = []
>            self._push(_cocall(main, *args, **kwds))
>
>        def resume(self, value = None):
>            return self._run(value, None)
>
>        def throw(self, exc):
>            return self._run(None, exc)
>
>        def close(self):
>            try:
>                self.throw(CoExit)
>            except (CoExit, CoReturn):
>                pass
>
>        def call(self, subroutine, *args, **kwds):
>            meth = getattr(subroutine, '__cocall__', None)
>            if meth is not None:
>                frame = meth(*args, **kwds)
>                if frame is not NotImplemented:
>                    self._push(frame)
>                    return self._run(None, None)
>            return CoReturn(subroutine(*args, **kwds))
>
>        #  Private methods
>
>        def _run(self, value, exc):
>            while True:
>                try:
>                    frame = self._top()
>                    if exc is None:
>                        return frame.__resume__(self, value)
>                    else:
>                        meth = getattr(frame, '__throw__', None)
>                        if meth is not None:
>                            return meth(self, exc)
>                        else:
>                            raise exc
>                except BaseException as exc:
>                    if self._pop():
>                        if isinstance(exc, CoReturn):
>                            value = exc.value
>                            exc = None
>                    else:
>                        raise
>
>        def _push(self, frame):
>            self._stack.append(frame)
>
>        def _pop(self):
>            if len(self._stack) > 0:
>                del self._stack[-1]
>                return True
>            else:
>                return False
>
>        def _top(self):
>            return self._stack[-1]
>
> --
> Greg
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>