Will Python 3.0 remove the global interpreter lock (GIL)

[Paul Rubin]

"Chris Mellon" <arkanes at gmail.com> writes:
> > The compute intensive stuff (image rendering and crunching) has
> > already had most of those skerricks pulled out.  It is written in C
> > and assembler
> That means that this part is also unaffected by the GIL.

Right, it was a counterexample against the "speed doesn't matter"
meme, not specifically against the GIL.  And that code is fast because
someone undertook comparatively enormous effort to code it in messy,
unsafe languages instead of Python, because Python is so slow.

> At the really high levels of scalability, such as across a server
> farm, threading is useless. The entire point of threads, rather than
> processes, is that you've got shared, mutable state. A shared nothing
> process (or Actor, if you will) model is the only one that makes sense
> if you really want to scale because it's the only one that allows you
> to distribute over machines. The fact that it also scales very well
> over multiple cores (better than threads, in many cases) is just
> gravy.

In reality you want to organize the problem so that memory intensive
stuff is kept local, and that's where you want threads, to avoid the
communications costs of serializing stuff between processes, either
between boxes or between cores.  If communications costs could be
ignored there would be no need for gigabytes of ram in computers.
We'd just use disks for everything.  As it is, we use tons of ram,
most of which is usually twiddling its thumbs doing nothing (as DJ
Bernstein put it) because the cpu isn't addressing it at that instant.
The memory just sits there waiting for the cpu to access it.  We
actually can get better-than-linear speedups by designing the hardware
to avoid this.  See:
  http://cr.yp.to/snuffle/bruteforce-20050425.pdf
for an example.

> The only hard example I've seen given of the GIL actually limiting
> scalability is on single server, high volume Django sites, and I don't
> think that the architecture of those sites is very scalable anyway.

The stuff I'm doing now happens to work ok with multiple processes but
would have been easier to write with threads.