Parallelization on muli-CPU hardware?

[Bryan Olson]

Alex Martelli wrote:

 > Aahz <aahz at pythoncraft.com> wrote:
 >>If you look at the code for _threading_local, you'll see that it depends
 >>on __getattribute__() to acquire a lock and patch in the current
 >>thread's local state.  IIRC from the various threads about securing
 >>Python, there are ways to break __getattribute__() [...]
 >
 > I believe you're thinking of the minimal portable version: I believe
 > that for such systems as Windows and Linux (together probably 99% of
 > Python users, and i speak as a Mac fan...!-) there are specific
 > implementations that use faster and probably unbreakable approaches.

Are we looking at the same question?  A language that makes safe
threading logically possible is easy; C does that. 'Unbreakable'
is a high standard, but one that a high-level language should
meet.  No matter how badly the Python programmer blows it,
Python itself should not crash, nor otherwise punt to arbitrary
behavior.

Now look at implementing Python without a global interpreter
lock (GIL).  To work safely, before we rely on any condition,
we must check that the condition is true; for example we do not
access an array element until we've checked that the subscript is
within the current array size.  Python does a reasonable job of
safety checking.  In the presence of OS-level multi-tasking,
Python could still fail catastrophically.  We might be suspended
after the check; then another thread might change the size of
the array; then the former thread might be restored before
accessing the element.

So what's the solution?  A global-interpreter-lock works; that's
what Python has now; but we lose much of the power of multi-
processing systems.   We could give every object its own lock,
but the locking overhead would defeat the purpose (and inter-
object conditions can be trickier than they might look).

Maybe every object, at any particular time, belongs to just one
thread.  That thread can do what it wants with the object, but
any other has to coordinate before accessing the object.  I
don't know the current results on that idea.

Functional programming languages point out a possible facility:
threads can safely share non-updatable values.  In Python,
ensuring that immutable objects never contain references to
mutable objects is a bigger change than we might reasonably
expect to impose.


Guido did not impose the GIL lightly.  This is a hard problem.


-- 
--Bryan