Embedding Python, threading and scalability

[Jeff Epler]

On Thu, Jul 10, 2003 at 11:14:47AM +0800, Simon Wittber (Maptek) wrote:
> Seriously though, this is an issue, which is a major hurdle Python *has*
> to cross if it is ever going to be seriously considered for use on large
> projects, on large SMP hardware.

Has anybody proposed "how to get there from here" for this problem
(useful multithreading of non-blocking pure Python code)?  I'm not
bright enough to see how, that's for sure.  Especially if you are
talking about an incremental approach, not the mythical "Python 3000".
What I mean is that you have to work this magic *and* somehow let
existing modules written in C work, with at worst a recompile.  (as
someone who doesn't *need* threads, but works on a project with piles of
Python modules written in C, that's my bias anyway)

Someone nearby mentioned lua, but I don't know what it did for threading.
Perl and tcl both seem to have taken the approach of having each thread
be a distinct interpreter with nothing shared.  While this means you
never have to worry about locking against a reader or modifier in another
thread, it means you might as well be using the processes that the angry
Unix Gods gave us in the first place. <1/3 overstatement>  I'm pretty sure
that this approach has been explicitly ruled out by that other angry God,
Guido van Rossum, anyway.

I've written Python programs that use threads in a way that was expedient
to get a user interface running without long seizures, while I've never
written a thread in tcl or perl.  OTOH if I'd had to treat everything
as explicit message passing (a la tcl threading), I'd have just found
another way (like "after idle" and "update" in tcl)

Jython will give you Java's thread model today, for Python code, won't
it?  Back when I benchmarked it, Jython code and Python code ran at
fairly similar speeds (in pybench), if only you had a machine that could
keep the Jython interpreter from swapping...

Jeff