Adding a Par construct to Python?

[jeremy at martinfamily.freeserve.co.uk]

On 19 May, 03:31, Carl Banks <pavlovevide... at gmail.com> wrote:
> On May 18, 1:52 pm, jer... at martinfamily.freeserve.co.uk wrote:
>
> > As I understand it the reason for the GIL is to prevent problems with
> > garbage collection in multi-threaded applications.
>
> Not really.  It's main purpose to prevent context switches from
> happening in the middle of some C code (essentially it makes all C
> code a "critical" section, except when the C code explicitly releases
> the GIL).  This tremendously simplifies the task of writing extension
> modules, not to mention the Python core.
>
> > Without it the
> > reference count method is prone to race conditions. However it seems
> > like a fairly crude mechanism to solve this problem. Individual
> > semaphores could be used for each object reference counter, as in
> > Java.
>
> Java doesn't use reference counting.
>
> Individual locks on the refcounts would be prohibitively expensive in
> Python, a cost that Java doesn't have.
>
> Even if you decided to accept the penalty and add locking to
> refcounts, you still have to be prepared for context switching at any
> time when writing C code, which means in practice you have to lock any
> object that's being accessed--that's in addition to the refcount lock.
>
> I am not disagreeing with your assessment in general, it would be
> great if Python were better able to take advantage of multiple cores,
> but it's not as simple a thing as you're making it out to be.
>
> Carl Banks

Hi Carl,

> I am not disagreeing with your assessment in general, it would be
> great if Python were better able to take advantage of multiple cores,
> but it's not as simple a thing as you're making it out to be.

Thanks for explaining a few things to me. So it would seem that
replacing the GIL with something which allows better scalability of
multi-threaded applications, would be very complicated. The paper by
Jesse Nolle which I referenced in my original posting includes the
following:

"In 1999 Greg Stein created a patch set for the interpreter that
removed the GIL, but added granular locking around sensitive
interpreter operations. This patch set had the direct effect of
speeding up threaded execution, but made single threaded execution two
times slower."

Source: http://jessenoller.com/2009/02/01/python-threads-and-the-global-interpreter-lock/

That was ten years ago - do you have any idea as to how things have
been progressing in this area since then?

Jeremy