I just killed GIL!!!

[Rhamphoryncus]

On Apr 17, 9:19 am, sturlamolden <sturlamol... at yahoo.no> wrote:
> On 17 Apr, 10:25, "Martin v. Löwis" <mar... at v.loewis.de> wrote:
>
> > help progress at all. I think neither was the case in this thread -
> > the guy claimed that he actually did something about the GIL, and
> > now we are all waiting for him to also tell us what it is that he
> > did.
>
> Ok, I did not remove the GIL, but I found a way to remove its
> notorious side effect on SMPs. So I am going to reveal it now. Forgive
> my strange sence of humor at 3 o'clock in the morning.
>
> First, think about this:
>
> (1) What is the GIL?
> (2) Where does it live?
> (3) How can it be manually released?
>
> The answer to this is:
>
> (1) A critical section (a lock/mutex object)
> (2) As a global object in Python25.dll on my computer
> (3) Using Python's C API or calling methods in a ctypes.CDLL object
>
> The Python C API has the ability to embed Python interpreters. You do
> this by importing Python25.dll into the process. ctypes has the
> ability to call functions in a DLL. So is it possible to embed Python
> in Python? And what would be consequence be?
>
> First, if I try to load a DLL more than once, Windows will detect this
> and just give me a handle to the currently imported library. So by
> importing Python25.dll with ctypes, I can just make my Python
> interpreter talk to itself through its own CAPI. Yes I can create sub
> interpreters using PyNew_Interpreter, but they all share the same GIL,
> so it's not useful here.
>
> So here is what I suddendly realized, and please don't laugh, its so
> simple its almost silly:
>
> I make some copies of Python25.dll, and call them Python25-1.dll,
> Python25-2.dll, Python25-3.dll, Python25-4.dll, etc. Then I load them
> all into my current Python process as ctypes.CDLL objects. Tada! I now
> have a pool of independent Python interpreters, not sharing the GIL,
> but still living in the same process.
>
> I can make the different interpreters talk to each other, including
> manipulating each other's GIL, using the using ctypes and Python's C
> API for embedding.
>
> So why does this circumvent the GIL? Because ctypes releases it before
> calling functions form a CDLL object.
>
> If I use my main interpreter to delegate a task to one of its embedded
> 'children', its GIL will be released while it is waiting for the
> answer. Associating each embedded interpreter with a threading.Thread
> is all that remains. The GIL is released while the thread operating
> the child interpreter is blocked.
>
> An there you have the answer. It's really very simple :-)

Interesting.  Windows specific, but there's other ways to do the same
thing more portably.

The bigger issue is that you can't share any objects.  This
effectively gives you a multiprocess model - a bit cheaper than that,
but not enough to really supply GIL-free threading.