Are the critiques in "All the things I hate about Python" valid?

[boB Stepp]

On Sat, Feb 17, 2018 at 12:54 AM, Chris Angelico <rosuav at gmail.com> wrote:
> On Sat, Feb 17, 2018 at 5:25 PM, boB Stepp <robertvstepp at gmail.com> wrote:

>>
>> I am curious as to what efforts have been attempted to remove the GIL
>> and what tradeoffs resulted and why?  Is there a single article
>> somewhere that collates this information?  I fear if I try to Google
>> this I will get a lot of scattered pieces that I will have to wade
>> through to get to what I want to know, where you (or someone else)
>> might be able to point me to a good link.  Or kindly summarize
>> yourself the relevant information.
>>
>> Thanks!
>
> No, there isn't a single article, at least not that I know of. A good
> word to search for is "gilectomy", which brought me to this talk by
> Larry Hastings:
>
> https://www.youtube.com/watch?v=pLqv11ScGsQ

Thanks for the link.  I'll give it a look tomorrow.  Must get some sleep soon!

Meanwhile I just finished reading "Efficiently Exploiting Multiple
Cores with Python" (from Nick Coghlan's Python Notes) at
http://python-notes.curiousefficiency.org/en/latest/python3/multicore_python.html
 It answered some of my questions and has a lot of good information.
It appears that a "gilectomy" as you put it is a really tough problem.
It looks to me that it might require a major
backwards-compatibility-breaking change to CPython to implement if I
am even close to understanding the issues involved.

> Broadly speaking, what happens is that removing a large-scale lock
> (the GIL) requires using a whole lot of small-scale locks. That gives
> finer granularity, but it also adds a whole lot of overhead; the CPU
> features required for implementing those locks are not fast. With the
> GIL, you claim it, and you can do what you like. Without the GIL, you
> have to claim a lock on each object you manipulate, or something along
> those lines. (Different attempts have gone for different forms of
> granularity, so I can't generalize too much here.) That means claiming
> and relinquishing a lot more locks, which in turn means a lot more
> CPU-level "lock" primitives. That's a lot of overhead.

Thanks for this explanation.  It really helps me a lot!

> One of the best ways to multi-thread CPU-intensive work is to push a
> lot of the work into an extension library. Take a function like this:
>
> def frobnosticate(stuff):
>     magic magic magic
>     magic magic more magic
>     return result
>
> As long as the returned object is a newly-created one and the
> parameter is not mutated in any way, you can do all the work in the
> middle without holding the GIL. That can't be done in pure Python, but
> in a C function, it certainly can. You still have the coarse-grained
> locking of the GIL, you still have all the protection, but you can now
> have two threads frobnosticating different stuff at the same time.

Yeah, Coghlan's article mentioned this.  I guess using Cython would be
one approach to this.

Thanks, Chris!


-- 
boB