[Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)

[Guido van Rossum]

On 9/17/05, John J Lee <jjl at pobox.com> wrote:
> Given the points you make, and the facts that both Python 3 and real
> problems with continuing to scale down semiconductor chip feature sizes
> are on the horizon, it seems that now would be an excellent time to start
> work on this, with the goal of introducing it at the same time as Python
> 3.
> 
> a. Python 3.0 will break lots of code anyway, so the extension module
>    issue becomes far less significant.
> 
> b. In x years time (x < 10?) it seems likely multiprocessor (MP) users
>    will be in the majority.  (As a result, the uniprocessor (UP) slowdown
>    becomes less important in practice, and also Python has the opportunity
>    of avoiding the risk of being sidelined by a real or perceived lack of
>    MP performance.)

That assumes a very specific model for how all that MP power is going
to be used. I personally don't think the threaded programming model as
found in Java works all that well; without locks you end up with
concurrent modification errors, with locks you get deadlocks and
livelocks. A typical programmer has a hard enough time keeping track
of a bunch of variables being modified by a single thread; add
multiple threads acting simultaneously on the same variables to the
mix, and it's a nightmare.

If my hunch is right, I expect that instead of writing massively
parallel applications, we will continue to write single-threaded
applications that are tied together at the process level rather than
at the thread level.

> c. Since time is needed to iron out bugs (and perhaps also to reimplememt
>    some pieces of code "from scratch"), very early in the life of Python 3
>    seems like the least-worst time to begin work on such a change.
> 
> I realize that not all algorithms (nor all computational problems) scale
> well to MP hardware.  Is it feasible to usefully compile both MP and a UP
> binaries from one Python source code base?

That's an understatement. I expect that *most* problems (even most
problems that we will be programming 10-20 years from now) get little
benefit out of MP.

> (I'm also quite aware that the GIL does not prevent all means of achieving
> efficient use of multiprocessors.  I'm just concious that different
> parellisation problems are presumably best expressed using different
> tools, and that Python 3 and increased prevalance of MP systems might tip
> the balance in favour of removing the GIL.)
> 
> Of course, it still takes a (anti-)hero to step forward and do the work...

Be my guest. Prove me wrong. Talk is cheap; instead of arguing my
points (all of which can be argued ad infinitum), come back when
you've got a working GIL-free Python. Doesn't have to be CPython-based
-- C# would be fine too.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)