Parallelization on muli-CPU hardware?

[Andrew Dalke]

P.M. wrote:
> My question is, how can I best parallelize the running of separate
> autonomous Python scripts within this app?  Can I run multiple
> interpreters in separate threads within a single process?

No.  Python only supports one interpreter per process.  There
may be several threads running, but only in the context of a
single interpreter.  Why?  Various data structures assume
global state.  One is support for different modules.

> In past
> newsgroup messages I've seen advice that the only way to get
> scalability, due to the GIL, is to use an IPC mechanism between
> multiple distinct processes running seperate interpreters.  Is this
> still true or are there better ways to accomplish this?

Hasn't changed.  There many ways to do it, ranging from
ZODB to Twisted to POSH to CORBA to shared memory to Pyro
to a few dozen more.  I've been waiting for a time to
give Pyro a try.

If most of your time is spent in Boost calculations then
you can release the GIL when you leave Python.  So long
as extensions do not call back into Python there won't
be a scaling problem due to the GIL.

				Andrew
				dalke at dalkescientific.com