[Python-Dev] Stackless Python - Pros and Cons

[Christian Tismer]

Jeremy Hylton wrote:
> 
> If someone is going to write a PEP, I hope they will explain how the
> implementation deals with the various Python C API calls that can call back
> into Python.

He will.

> In the stackless implementation, builtin_apply is a thin wrapper around
> builtin_apply_nr.  The wrapper checks the return value from builtin_apply_nr
> for Py_UnwindToken.  If Py_UnwindToken is found, it calls
> PyEval_Frame_Dispatch. In this case, builtin_apply returns whatever
> PyEval_Frame_Dispatch returns; the frame dispatcher just executes stack
> frames until it is ready to return.

Correct.

> How does this control flow at the C level interact with a Python API call
> like PySequence_Tuple or PyObject_Compare that can start executing Python
> code again?  Say there is a Python function call which in turn calls
> PySequence_Tuple, which in turn calls a __getitem__ method on some Python
> object, which in turn uses a continuation to transfer control.  After the
> continuation is called, the Python function will never return and the
> PySquence_Tuple call is no longer necessary, but there is still a call to
> PySequence_Tuple on the C stack.  How does stackless deal with the return
> through this function?

Right. What you see here is the incompleteness of Stackless.
In order to get this "right", I would have to change many
parts of the implementation, in order to allow for continuations
in every (probably even unwanted) place.
I could not do this.

Instead, the situation of these still occouring recursions
are handled differently. continuationmodule guarantees, that
in the context of recursive interpreter calls, the given
stack order of execution is obeyed. Violations of this
cause simply an exception.

> I expect that any C function that may cause Python code to be executed must
> be wrapped the way apply was wrapper.  So in the example, PySequence_Tuple
> may return Py_UnwindToken.  This adds an extra return condition that every
> caller of PySequence_Tuple must check.  Currently, the caller must check for
> NULL/exception in addition to a normal return.  With stackless, I assume the
> caller would also need to check for "unwinding."

No, nobody else is allowed to return Py_UnwindToken but the few
functions in the builtins implementation and in ceval. The
continuationmodule may produce it since it knows the context
where it is called. eval_code is supposed to be the main instance
who checks for this special value.

As said, allowing this in any context would have been a huge
change to the whole implementation, and would probably also
have broken existing extensions which do not expect that
a standard function wants to do a callback.

> Is this analysis correct? Or is there something I'm missing?
> 
> I see that the current source release of stackless does not do anything
> special to deal with C API calls that execute Python code.  For example,
> PyDict_GetItem calls PyObject_Hash, which could in theory lead to a call on
> a continuation, but neither caller nor callee does anything special to
> account for the possibility.  Is there some other part of the implementation
> that prevents this from being a problem?

This problem is no problem for itself, since inside the stackless
modification for Python, there are no places where unexpected
Py_UnwindTokens or continuations are produced. This is a closed
system insofar. But with the continuation extension, it is
of course a major problem.

The final solution to the recursive interpreter/continuation
problem was found long after my paper was presented. The idea
is simple, solves everything, and shortened my implementation
substantially:

Whenever a recursive interpreter call takes place, the calling
frame gets a lock flag set. This flag says "this frame is wrapped
in a suspended eval_code call and cannot be a continuation".
continuationmodule always obeys this flag and prevends the
creation of continuations for such frames by raising an
exception. In other words: Stack-like behavior is enforced
in situations where the C stack is involved.

So, a builtin or an extension *can* call a continuation, but
finally, it will have to come back to the calling point.
If not, then one of the locked frames will be touched,
finally, in the wrong C stack order. But by reference
counting, this touching will cause the attempt to create
a continuation, and what I said above will raise an exception.

Probably the wrong place to explain this in more detail here, but
it doesn't apply to the stackless core at all which is just
responsible for the necessary support machinery.

ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH      :     Have a break! Take a ride on Python's
Kaunstr. 26                  :    *Starship* http://starship.python.net
14163 Berlin                 :     PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 93E2 1FAE F6DF
     where do you want to jump today?   http://www.stackless.com