Stackless Platform Independence?

[Frederic Giacometti]

Skip Montanaro wrote:

>     Frederic> Just reverse the paradigm: have the python VM run in a single
>     Frederic> thread, using stackless microthreads for python
>     Frederic> multithreading, and run the C calls in a pool of threads.
>     Frederic> This way: no lock is needed, since only one thread runs the
>     Frederic> VM, with the help of microthreads, and C calls run in other
>     Frederic> threads.
>
> I don't think this will work without a lot of effort.  There is code that is
> not strictly speaking part of the virtual machine that assumes the presence
> of the global interpreter lock.

It can perfectly cohabitate with the current thread implementation; there is no
incompatibility.

An implementation would begin with defining a new CFunction flag (the 3rd field
of the CFunction data structure.
This flag would have the PVM instead call a function that would:
  - remove the microthread from the queue of active microthreads
  - pick an OS thread from the thread pool, place the CFuntion and its args in
its exec queue, and return to the PVM

Upon return of the CFunction, the OSThread would put back the thread on the
queue of active threads of the PVM, along with its result object, "et le tour
est joue'".

The C programmation of this new type of CFunction would be a little bit
different:
1) on one hand, no global interpreter lock to release :))
2) on another hand, by default, nothing should be done that modifies the
reference count of PyObjects referenced in other OS threads (note: the OS thread
can however create plain new Python objects); but this is actually no different
that current Python C programming after releasing the Python lock.

Helper functions/macros could provided to circumvent 2):
  - either recourse to the global Python lock (nothing to invent/implement,
here)
  - create a function that create a microthread that will have the PVM
increment/decrement the count(s) (and maybe to something else too), place it on
the PVM queue, and wait synchronously until it returns.

There is a third venue too: Add a field to the PyObject structure that describes
'OS Thread ownership' (the 0 value would certainly refer to the master PVM).
Then:
  - Rule: OS threads can only modified reference count of objects they own
  - Helpers: provide functions that gives the possibility to an OS thread to ask
the thread that owns the object(s), to transfer ownership. Such functions would
be used whenever local ownership is not certain (there's no point in testing
ownership on each reference counting operation) ....

Technically, this addition to the PyObject structure would open the door to:
   - multiple PVM on multiprocessing machines
   - remote Python object transparency

Such a schema would require some elaboration, like any; but there's no wizardry
here, and it could be done with a minimal imprint on the current PVM code, too.

But:
  - microthreads are a preamble
  - stackless is a preamble to microthreads...

Frederic Giacometti, Arakne

>
>
> --
> Skip Montanaro (skip at pobox.com - http://www.mojam.com/)