Stackless Python, eventual merge?

[Christian Tismer]

Michael Schneider wrote:
...

>    "stackless python behaves the same as 'normal' python, until you 
> import the
>    stackless module.  At that point, it becomes stackless python.  This 
> allows you
>     to run 'stackless unsafe' modules with the same interpretor as 
> stackless modules,
>     in a predictable manner (as long as you don't import stackless, and 
> an unsafe module)"
> 
> 
> Is this the correct understanding????

Almost. It is slightly different (and slightly better):

Part I: stack slicing
---------------------
Stackless Python has built-in support for stack slicing.
This is by default off. Stackless Python behaves exactly
like Python as long as you don't activate things.

There is a couple of new functions in the sys module
which control behavior:
old_flag = sys.enable_stackless(new_flag)
sets stackless behavior to on or off and returns the
old state.
When it is enabled, then the C stack will be hijacked
every nth interpreter recursion, in order so save
stack space. This behavior is controlled by the two
functions
sys.setslicinginterval(n)
sys.getslicinginterval()

If this interval is set very high, you will also get no
stack slicing.

Part II: Multitasking
---------------------
Completely apart from the features of part I, there is
an extension module named stackless.
It can be imported only if the Python interpreter was built
for it, since it does in fact make use of the above
stack slicing technique.
But instead of slicing stacks in order to save space
in deep recursion, the stackless module uses it to
move stacks forth and back. This is like switching threads,
just not by stack switching but moving.
These multitasking features are independant from the
sys options of part I. If a tasklet is asked to switch
context to another one, it will do that, since you demanded
it. There will be no switching if you don't ask for it.

Both parts can be used in any combination.
If small stack size is desirable, turn stack slicing on.
If you want to use tasklets, use them.
In combination, you will of course xperience the best
performance.

About extension modules:
------------------------
Stack slicing has shown up as a problem with tkinter.
I had to modify the _tkinter module to make this safe
(and I really should include that in the distro now).
The problem with stack slicing is that it occours in
a context where an extension might expect certain
data to be still on the C stack.

Usually, using tasklets is much more safe than stack
slicing. It depends on your application. But a tasklet
switch, while technically the same, has the semantics
of a context switch. You are moving out of a context
where some extensiom module might be involved, into
a different context, where this extension module
is not active.

But still some care needs to be taken, since tasklets
do microthreading, which is scheduling at a finer grain
than real threads. An extension module (or even Python
module) might be thread safe, but not microthread safe.
The reason is that the extension does not have any knowledge
about this context switching, it still thinks of the
one existing thread.
For that reason, one should avoid to use certain
resources more than once in a real thread. Some
locking should be used to protect data structures.

At the moment the above problems are rather easy to
handle, since context switching is explicitly done
by the user.
I am going to support pre-emptive scheduling, soon,
which makes things more difficult, and I will provide
a mechanism that inhibits auto-scheduling unless
the user declares a module as definativly microthread
safe.

so much so far - ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer at tismer.com>
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34  home +49 30 802 86 56  pager +49 173 24 18 776
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/