Fake threads (was [Python-Dev] ActiveState & fork & Perl)

[Christian Tismer]

Just a few clarifications.

I have no time, but need to share what I learned.

Tim Peters wrote:
> 
> [Guido and Tim, Guido and Tim]
...

> Among the three {coroutines, fake threads, continuations}, I expect any
> could be serviceably simulated via either of the others.  There:  just saved
> a full page of sentence diagramming <wink>.  All offer a strict superset of
> generator semantics.

I have just proven that this is not true. Full continuations
cannot be expressed by coroutines. All the rest is true.

Coroutines and fake threads just need the absence of the
C stack. To be more exact: It needs that the current state
of the C stack is independent from executing bound Python code
(which frames are).
Now the big surprize: This *can* be done without removing
the C stack. It can give more speed to let the stack wind
up to some degree and wind down later. Even some Scheme
implementations are doing this. 
But the complexity to make this work correctly is even higher
than to be stackless whenever possible. So this is the basement,
but improvements are possible and likely to appear.

Anyway, with this, you can build fake threads, coroutines
and generators. They all do need a little extra treatment.
Switching of context, how to stop a coroutine, how to
catch exceptions and so on. You can do all that with some
C code. 
I just believe that even that can be done with Python.
Here the unsayable continuation word appears. You must
have them if you want to try the above *in* Python.

Reason why continuations are the hardest of the above to
implement and cannot expressed by them:
A continuation is the future of some computation. It allows
to change the order of execution of a frame in a radical
way. A frame can have as many as one dormant continuation
per every function call which appears lexically, and it
cannot predict which of these is actually a continuation.

From that follows: The Python stack, which keeps intermediate
values under the (now wrong) assumption that it knows the
execution order, is in the way. Just think of a for loop,
but I found other examples where this is the case.
Simple stack capturing does just half the job.

Conclusion: In order to have full continuations (and believe
me, just to try it, without pushing to get it into Python),
one needs to identify the stack locations which carry
values which are needed in more than one continuation.
In other words, we have to find hidden registers.

I discovered this rather late when I claimed to have them
already, since my examples all worked fine. And in most cases,
this will stay so. Just to make every situation consistent,
the above extra analysis is a must, IMO.

This is my path and the end of this journey:
- Refine stackless Python to be strong enough to even
  stand continuations. This is done.
- Write the whole co-anything stuff as a pure extension module.
  Also done. Without this extension, Python is *not* affected.
- Solve the last issue of shared hidden registers. I wrote
  the analyzer in C already. Now I need time to do the 
  wind/unwind stuff. Also in the extension, no change to Python!

When this is done, what do we have?
We have Python with continuations, where I'm not sure if that
is what we finally want. No question that this is an issue.
BUT: We can now implement the real things like fake-threads,
coroutines, generators and whatever else we come up with,
given these continuations.

I hope every listener sees what I really want: When the above
sandbox is there, then we can try a lot and find the best,
nicest, easiest to grasp, most natural and straight forward
way to implement the high level functions and objects.
When that is done, and not before, then we should code
"the right thing"(TM) in C, and we can forget about
continuations if nobody needs them any longer.

Sorry that my path got longer than I planned.
I cannot understand certain things without implementing them.

sincerely - chris

-- 
Christian Tismer             :^)   <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH      :     Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101   :    *Starship* http://starship.python.net
10553 Berlin                 :     PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 93E2 1FAE F6DF
     we're tired of banana software - shipped green, ripens at home