Python threading (was: Re: global interpreter lock not working as it should)

[Jonathan Hogg]

On 3/8/2002 2:43, in article aifcfh$jha$0 at 216.39.172.122, "Bengt Richter"
<bokr at oz.net> wrote:

>> This is true. Therefore, the only time another thread WILL grab the GIL under
>> Linux is if 
>> 
>> 1) the GIL is released by the currently running thread
>> 2) the thread that just released the GIL depletes its timeslice before it
>>   can grab the lock again
>> 3) the OS notices the process has depleted it's timeslice and the yanks
>>   it from the CPU (this happens every 100 times per second by default on an
>>                    i386)
>> 4) the waiting thread that recieved the GIL release signal is chosen to run
>> 
>> Therefore, we now have a large set of coincidences for CPU-bound python
>> threads. The only reason it works at all is because it happens 100 times per
>> second and the GIL is released frequently by default. So there is a
>> sufficient probability that these 4 cases will happen simultaneously.
>> 
> Hm. If that's an accurate description (and I am skeptical ;-), I don't think I
> like it.
> Seems like if another thread is waiting for the GIL, then after a full time
> slice the
> current GIL-holder ought to be told to give it up ASAP and be given a new time
> slice
> only to do the minimum it needs to do that.

If this is true, and I'm also skeptical, then it's a bug in Linux not Python
and Python should do nothing about it.

I ran a little experiment (code enclosed below) to crudely check the thread
switching on my machine (Mac OS X - well known for it's threading). Running
10 CPU-bound threads together for 10 seconds came up with them all getting
fairly even CPU time (within +/- 10% of the mean).

My crude count for the actual number of thread switches that occurred during
those 10 seconds showed never less than 1200 thread switches per second and
up to 3000 thread switches per second.

I also tried the same code on a FreeBSD, Solaris, and Linux box. The summary
results were:

    OS               Architecture   Python     Thread Switches (avg 2 runs)
    Mac OS X 10.1    PPC            2.3-CVS    24971
    FreeBSD 4.3      i386           2.2.1      158499
    Linux 2.2.14     i386           2.2        18281
    Solaris 8        SPARC          2.2.1      243

So an interesting spread with Solaris being way down there in terms of how
often it switched threads (though the CPU distribution seemed to be pretty
fair anyway) - I'm not sure what threading library Python was compiled
against on the Solaris box (which has two - one OS and one user), I might
look into that out of curiosity. Anyone know FreeBSD well enough to comment
on the results for that platform?

Note that Linux is still doing over 1800 switches per second, which makes me
even more skeptical that it is only capable of thread switching due to a
huge coincidence.

Any comments?

Jonathan


----- threads.py -----
import threading
import time
import operator

def sum( xs ):
    return reduce( operator.add, xs )

QUITTING = 0
RUNNING = None

class Counter( threading.Thread ):
    def run( self ):
        global QUITTING, RUNNING
        count = 0
        switches = 0
        while not QUITTING:
            count += 1
            if self is not RUNNING:
                RUNNING = self
                switches += 1
        self.count = count
        self.switches = switches

counters = [ Counter() for i in range(10) ]

for counter in counters:
    counter.start()

time.sleep( 10.0 )

QUITTING = 1
for counter in counters:
    counter.join()

counts = [ counter.count for counter in counters ]
switches = [ counter.switches for counter in counters ]

print "Counts:"
print counts
print "Total =", sum( counts )

print "\nSwitches:"
print switches
print "Total =", sum( switches )