Thread scheduling

[Jack Orenstein]

I am using Python 2.2.2 on RH9, and just starting to work with Python
threads.

I started using the threading module and found that 10-20% of the runs
of my test program would hang. I developed smaller and smaller test
cases, finally arriving at the program at the end of this message,
which uses the thread module, not threading. This program seems to
point to problems in Python thread scheduling.

The program is invoked like this:

     python threadtest.py THREADS COUNT

THREADS is the number of threads created. Each thread contains a loop
that runs COUNT times, and all threads increment a counter. (The
counter is incremented without locking -- I expect to see a final
count of less than THREADS * COUNT.)

Running with THREADS = 2 and COUNT = 100000, most of the time, the
program runs to completion. About 20% of the time however, I see one
thread finish, but the other thread never resumes.

Here is output from a run that completes normally:

     [jao at black]$ python threadtest.py 2 100000
     nThreads: 2
     nCycles: 100000
     thread 1: started
     thread 1: i = 0, counter = 1
     thread 2: started
     thread 2: i = 0, counter = 2691
     thread 1: i = 10000, counter = 13496
     thread 2: i = 10000, counter = 22526
     thread 1: i = 20000, counter = 27120
     thread 2: i = 20000, counter = 40365
     thread 1: i = 30000, counter = 41264
     thread 1: i = 40000, counter = 55922
     thread 2: i = 30000, counter = 58416
     thread 2: i = 40000, counter = 72647
     thread 1: i = 50000, counter = 74602
     thread 1: i = 60000, counter = 88468
     thread 2: i = 50000, counter = 99319
     thread 1: i = 70000, counter = 110144
     thread 2: i = 60000, counter = 110564
     thread 2: i = 70000, counter = 125306
     thread 1: i = 80000, counter = 129252
     Still waiting, done = 0
     thread 2: i = 80000, counter = 141375
     thread 1: i = 90000, counter = 147459
     thread 2: i = 90000, counter = 155268
     thread 1: leaving
     thread 2: leaving
     Still waiting, done = 2
     All threads have finished, counter = 168322

Here is output from a run that hangs. I killed the process using
ctrl-c.

     [jao at black]$ python threadtest.py 2 100000
     nThreads: 2
     nCycles: 100000
     thread 1: started
     thread 1: i = 0, counter = 1
     thread 2: started
     thread 2: i = 0, counter = 990
     thread 1: i = 10000, counter = 11812
     thread 2: i = 10000, counter = 13580
     thread 1: i = 20000, counter = 19127
     thread 2: i = 20000, counter = 25395
     thread 1: i = 30000, counter = 31457
     thread 1: i = 40000, counter = 44033
     thread 2: i = 30000, counter = 48563
     thread 1: i = 50000, counter = 55131
     thread 1: i = 60000, counter = 65291
     thread 1: i = 70000, counter = 78145
     thread 2: i = 40000, counter = 82715
     thread 1: i = 80000, counter = 92073
     thread 2: i = 50000, counter = 101784
     thread 1: i = 90000, counter = 104294
     thread 2: i = 60000, counter = 112866
     Still waiting, done = 0
     thread 1: leaving
     Still waiting, done = 1
     Still waiting, done = 1
     Still waiting, done = 1
     Still waiting, done = 1
     Still waiting, done = 1
     Still waiting, done = 1
     Still waiting, done = 1
     Still waiting, done = 1
     Traceback (most recent call last):
       File "threadtest.py", line 26, in ?
         time.sleep(1)
     KeyboardInterrupt
     [jao at black osh]$

In this case, thread 1 finishes but thread 2 never runs again. Is
this a known problem? Any ideas for workarounds? Are threads widely
used in Python?

Jack Orenstein



# threadtest.py

import sys
import thread
import time

nThreads = int(sys.argv[1])
nCycles = int(sys.argv[2])
print 'nThreads: %d' % nThreads
print 'nCycles: %d' % nCycles
counter = 0
done = 0

def run(id):
     global done
     print 'thread %d: started' % id
     global counter
     for i in range(nCycles):
         counter += 1
         if i % 10000 == 0:
             print 'thread %d: i = %d, counter = %d' % (id, i, counter)
     print 'thread %d: leaving' % id
     done += 1

for i in range(nThreads):
     thread.start_new_thread(run, (i + 1,))
while done < nThreads:
     time.sleep(1)
     print 'Still waiting, done = %d' % done
print 'All threads have finished, counter = %d' % counter