I just killed GIL!!!

[sturlamolden]

On 17 Apr, 10:25, "Martin v. Löwis" <mar... at v.loewis.de> wrote:

> help progress at all. I think neither was the case in this thread -
> the guy claimed that he actually did something about the GIL, and
> now we are all waiting for him to also tell us what it is that he
> did.


Ok, I did not remove the GIL, but I found a way to remove its
notorious side effect on SMPs. So I am going to reveal it now. Forgive
my strange sence of humor at 3 o'clock in the morning.

First, think about this:

(1) What is the GIL?
(2) Where does it live?
(3) How can it be manually released?

The answer to this is:

(1) A critical section (a lock/mutex object)
(2) As a global object in Python25.dll on my computer
(3) Using Python's C API or calling methods in a ctypes.CDLL object

The Python C API has the ability to embed Python interpreters. You do
this by importing Python25.dll into the process. ctypes has the
ability to call functions in a DLL. So is it possible to embed Python
in Python? And what would be consequence be?

First, if I try to load a DLL more than once, Windows will detect this
and just give me a handle to the currently imported library. So by
importing Python25.dll with ctypes, I can just make my Python
interpreter talk to itself through its own CAPI. Yes I can create sub
interpreters using PyNew_Interpreter, but they all share the same GIL,
so it's not useful here.

So here is what I suddendly realized, and please don't laugh, its so
simple its almost silly:

I make some copies of Python25.dll, and call them Python25-1.dll,
Python25-2.dll, Python25-3.dll, Python25-4.dll, etc. Then I load them
all into my current Python process as ctypes.CDLL objects. Tada! I now
have a pool of independent Python interpreters, not sharing the GIL,
but still living in the same process.

I can make the different interpreters talk to each other, including
manipulating each other's GIL, using the using ctypes and Python's C
API for embedding.

So why does this circumvent the GIL? Because ctypes releases it before
calling functions form a CDLL object.

If I use my main interpreter to delegate a task to one of its embedded
'children', its GIL will be released while it is waiting for the
answer. Associating each embedded interpreter with a threading.Thread
is all that remains. The GIL is released while the thread operating
the child interpreter is blocked.

An there you have the answer. It's really very simple :-)



Regards,

Sturla Molden