Using python as primary language

[Chris Mellon]

On Nov 9, 2007 9:54 AM, Martin Vilcans <martin at librador.com> wrote:
>
> On Nov 9, 2007 10:37 AM, Hrvoje Niksic <hniksic at xemacs.org> wrote:
> > "Martin Vilcans" <martin at librador.com> writes:
> >
> > >> If by 'this' you mean the global interpreter lock, yes, there are good
> > >> technical reasons.  All attempts so far to remove it have resulted in an
> > >> interpeter that is substantially slower on a single processor.
> > >
> > > Is there any good technical reason that CPython doesn't use the GIL
> > > on single CPU systems and other locking mechanisms on multi-CPU
> > > systems?
> >
> > It's not the locking mechanism itself that is slow, what's slow is the
> > Python you get when you remove it.  By removing the GIL you grant
> > different threads concurrent access to a number of shared resources.
> > Removing the global lock requires protecting those shared resources
> > with a large number of smaller locks.  Suddenly each incref and decref
> > (at the C level) must acquire a lock, every dict operation must be
> > locked (dicts are used to implement namespaces and in many other
> > places), every access to a global (module) variable must be locked, a
> > number of optimizations that involve using global objects must be
> > removed, and so on.  Each of those changes would slow down Python;
> > combined, they grind it to a halt.
>
> But if Python gets slow when you add fine-grained locks, then most
> certainly it wouldn't get so slow if the locks were very fast, right?
>
> But that's not what my question was about. It was about whether it
> would make sense to, on the same python installation, select between
> the GIL and fine-grained locks at startup. Because even if the locks
> slows down the interpreter, if they let you utilize a 32 core CPU, it
> may not be so bad anymore. Or am I missing something?
>

The performance overhead of a runtime swap between fine-grained locks
and the GIL would probably swamp any benefit you gained from having it
be switchable. A compile time switch is more reasonable, but it has
all the other problems of going fine-grained in the first place (it
breaks compatibility with existing extensions, large code size
increase, increased maintenance burden) and it adds the additional
problem that one or other of the modes won't be as heavily tested
(this will start out as the fine-grained mode, of course, which is the
worst case because it's the more complicated and subtly error prone
one). It's really best in the long term to pick one and stick with it.

The heavy use of dicts is one of the problems - they're all over the
place and even if you removed the GIL, a "global dict lock" would give
essentially the same effect. And a per-dict lock means that there will
be a *lot* more locking and unlocking, which means a slower
interpreter.