Microthreads without Stackless?

[Bryan Olson]

David Mertz, Ph.D. wrote:
 > Well, a couple things.  The article on (semi-)coroutines is really a
 > different topic than the one on "weightless threads."  They both take
 > advantage of Python generators.  And they both involve a scheduler.
 > But there are differences between the techniques... and still greater
 > differences between the motivations that would prompt the use of each.
 >
 > Unfortunately, Bryan started the discussion with a claim like
 > "weightless threads aren't *really* coroutines"... to which the answer
 > is "Duh!"

That's not a quote of me.  What I did say is, "Mertz's pattern
provides nothing like real co-routines or threads".  By "real"
coroutines I've meant coroutines without the limitations stated
for semi-coroutines.

 > Weightless threads basically amount to "cooperative multitasking."
 > Unlike in actual Stackless (or with "weighty" OS threads), the
 > weightless thread schedulers/technique depends every "thread"
 > (procedure) acting nice, and giving control back to the scheduler
 > without too much delay.

Again, I have to disagree with the first sentence there.  I've
never heard "Cooperative multitasking" to refer to calls that
were limited to a one-level depth.

I find the usual description of the difference between
cooperative and preemptive multi-threading to be deceptive. In
either system, the vast majority of thread switches are induced
by the running thread beginning an operation that blocks.



 > Some older OS's were written this way, FWIW,
 > but those had obvious problems.

Microsoft Windows and Apple Mac-OS had cooperative multi-tasking
when they took over most of the world.  They certainly had
problems, but that intra-process threading was cooperative was
not high on the problem list.  (The way separate processes could
effect each other, that was a problem.)


[...]
 > At the Python Cookbook site, Bernhard Mulder unfortunately posted a
 > recipe that he called "coroutines", when what it really does is
 > implement weightless threads.  The recipe itself is fine, but I hate
 > to see people mislead on the nomenclature:
 >
 >   http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/300019
 >

The point I got from Bernhard Mulder's recipe is how to yield
from any call depth.  His method is to re-write every call chain
that leads to a yield, so that every function is a generator and
every call is in a for loop.  With patterns of the form 'rewrite
everything like this...', any Turing-complete language can
emulate any computational facility of any other language.
That's the kind of solution I was discounting when I wrote,
"Importantly, I do *not* have to re-write all the functions in
every call chain that leads to a read or write".


 > Now it's true that Python syntax -in itself- only implements
 > semi-coroutines.  That is, you can branch INTO any procedure you want,
 > but the branch OUT OF that procedure is still stack like... i.e. to
 > the caller only.  What my article points out (which a lot of people
 > did not realize when Python generators were new) is that
 > semi-coroutines are actually fully general.

Syntax isn't the issue.  Python was just as general without
generators, which we can emulate with closures or classes.

[...]
 > The only change is that you always "yield to" a given procedures,
 > rather than either call it, or call its .next() method.  Of course,
 > lots of other flows are possible other than stack-like.  But you are
 > not prohibited from stack-like flow (example typed without testing,
 > forgive any minor typo):
 >
 >   from __future__ import generators
 >   cargo = None
 >   def grandma():
 >       yield (MOM, cargo)
 >   def mom():
 >       yield (DAUGHTER, cargo)
 >       yield (GRANDMA, cargo)
 >   def daughter():
 >       yield (MOM, cargo)
 >   GRANDMA, MOM, DAUGHTER = grandma(), mom(), daughter()
 >   scheduler(GRANDMA)

Again, that's a 'rewrite every call-chain' solution.  If we look
at the socket servers in the Python library, they're written on
a base where one server handles one client.  Then there's a
forking mix-in, and a threading mix-in, that each allow the same
code to handle many clients.  We can build higher-level
protocols on them, with the same simple form: handle one client,
and add forking/threading to handle many clients, without re-
writing the code.  Full coroutines or micro-threads can offer
the same facility.  Semi-coroutines or "weightless threads" are
not powerful enough to do that.


--
--Bryan