Stackless, thread paradigm and C extensions

[Frederic Giacometti]

"Paul Rubin" <phr-n2001d at nightsong.com> wrote in message
news:7x8zditn72.fsf at ruckus.brouhaha.com...
> Yes, it's a standard idiom of continuations to implement coroutines
> with them.
>
> One virtue of using kernel threads is in principle the kernel threads
> can take advantage of multiple hardware CPU's.  Python's global
> interpreter lock stops that from happening, but I thought there was
> some hope the lock might go away someday.

Here are more details of a possible design of the stackeless OS thread
model:

[Preamble: In a single thread build, all this collapses into one thread]

The Python VM thread (this thread that python at <<full speed>>, without
interruption) is associated to two pools of OS threads.

The thread in the first pool runs arbitrary reentrant C functions.
The threads in the second pool are each associated to a non-reentrant C
API's (one thread per independent sets of API; to be initialized upon
loading the extension module); e.g. OpenGL, Metaphase...

Each thread schedules its own queue of continuations (micro-threads) for
execution.
The C threads can post new continuations to the Python thread, and
vice-versa.

In total, three main types of continuations are used:
   - continuations from the Python thread, scheduled in the Python thread
   - continuations from C function threads, scheduled in the Python thread
   - continuations from the Python thread, scheduled in the C threads

 This model seems generic; it has no interpreter lock, and can simply
distribute C calls across multiple OS threads. Multithreaded C/Python
programming would be simplified too.

[As an exemple, with this model, in JPE, the Java threads would be mapped to
python continuations; and this would work without regard whether the JVM
uses native or green threads.
In contrast, without stackless, JPE requires that the JVM runs with native
thread, and has to bear the constant thread switching/Python lock overhead
of the present implementation...].

Furthermore, non-blocking versions of some of the Python C functions can be
provided.
For instance, the functions py_decref() and py_decref_non_blocking() could
be provided....

Finally, the concept can be pushed as far as dedicating a third thread to
python de-referencing and memory de-allocation, that would run in parallel
to the python thread; thus releasing the later from the load of
de-referencement.

FG