[Python-Dev] Re: [python-coro] coroutines and microthreads

[Guido van Rossum]

(A typically brotherly dialog. :-)

[Guido]
> >This is about as simple as it gets.  It supports both generators (like
> >the example here) and coroutines (like Tim's Demo/threads/squasher.py
> >example).

[Just]
> Neat! (It's _very_ close to what I proposed myself, so how can I not like
> it? ;-)

I looked at what you did, and probably a dozen other examples.

> One question: I assume values can also be passed when explicitly resuming a
> coroutine. However, the _first_ time a coroutine is "resumed" (started,
> really), the stored init arguments get passed to the function. What to do
> with a value passed to the resume call? Two possible solutions:
> 1) don't not allow passing a value the first time
> 2) add the value(s) to the init arguments
> 
> As code:
> 
>   def fribble(initarg):
>      ...
> 
>   c = coroutine(fribble, initarg)
>   c(x)  # <-- where does x go? or is it not allowed?

According to Tim, it's a standard issue that every coroutine API has
to deal with.  I lay awake worrying about this for a long time.  I
read the thread about this in the python-coro list.  I even started
thinking about a very different API, that would disinguish between
get() and put() operations.  But there was nothing that satisfied all
constraints.  My current solution: for the initial transfer, *no
argument is allowed*.  (Except None. :-)  I've looked at several
examples (8 queens, squasher, fringe) and they all work fine under
this assumption.

> >Besides the APIs shown here (coroutine(), suspend(), and EarlyExit) I
> >propose a function current() which returns the current coroutine
> >object.
> 
> Starts looking more and more like what I proposed myself...

Great minds and all that... :-)

I've been thinking about names a bit more, and 'main' is a truly lousy
name.  I now propose a function initial() to get at the main
coroutine.  (A function out of symmetry with current(), but also
because in an environment that also supports OS-level threads, there
is a different main coroutine per OS thread.)

> >There should also be a way to kill a coroutine (or at least
> >to send an exception).  When a coroutine falls through at its end,
> >*some* other coroutine needs to be resumed.
> 
> Here's what I did in my own strawman:
> - resume the guy that resumed me, if any (making it equivalent to suspend())

But there could be a cycle here!  What if A resumed B resumed C
resumed A?

> - if we were resumed by a suspend() call (which in my proposal means we
> don't have a "resumer", to avoid endless suspend()/suspend() bouncing),
> resume the "main" coroutine, which is the (non-coro) code that started the
> first coroutine.

But the main coroutine may not be expecting this -- e.g. in the
fringe() example, the main coroutine is expecting the next fringe
item or an EarlyExit exception.

> Maybe coroutines should have a kill() method, which would post the
> EarlyExit exception. Maybe even a postException() method (like uthreads
> have), allowing you to post arbitrary exceptions to the coroutine. Dunno.

Indeed.  This needs a bit more thinking (which so far I've
successfully managed to avoid :-).

> >I believe this can be implemented with a much simplified stackless
> >approach, that doesn't cater towards continuations (but still borrows
> >a lot of wisdom from Christian's Stackless).  It can also be
> >implemented using threads, which should give some hope for getting the
> >same API supported in JPython, making it more attractive.  I am hoping
> >to create an implementation on top of Stackless, just to experiment
> >with the idiom.
> 
> Here's my own code, based on stackless 1.1:
>    http://www.petr.com/just/corojust.tar.gz
> or
>    http://www.petr.com/just/corojust/
> for the individual files.
> 
> Apart from the initialisation it's pretty much what you propose.

I saw that, and even peeked a bit at it.  It seems you must have
struggled a bit with the initialization sequence in the squasher
example, gmcm_coro.py (which BTW is due to Tim Peters in its current
-- Python -- form).  That's alright -- coroutine initialization *is* a
problem.

I've found that finalization is a problem, too: I'm not at all sure
that my version of that example correctly frees all the coroutines
when it is done, since several of them are still suspended (waiting
for more input) when the decision to exit is made.  Also, I had to
invent the 'main' API in order to make it work.  Tim's original
(Demo/threads/squasher.py in the CVS tree) has a kill operation that
makes this easier.  His kill() kills the entire *group* of related
coroutines -- a concept that we don't have and probably don't need: it
just makes using the API more complicated, and Tim has said he'd do
away with it if he had to do it again.

BTW, I've got an almost-working implementation of the coro API using
Stackless continuations!  Problem: I don't know how to raise an
exception for a continuation -- which I need to implement EarlyExit.
Can anyone help?

--Guido van Rossum (home page: http://www.python.org/~guido/)