How to increase number of threads per process?

[Steve]

Ronan,

Ronan Viernes wrote:
> Hi,
> 
> I have created a python script (see below) to count the maximum number
> of threads per process (by starting new threads continuously until it
> breaks).
> 
> ######
> #testThread.py
> import thread, sys
> 
> def main():
>     print "Main Thread:", thread.get_ident()
>     count = 0;
>     try:
>         while 1:
>             thread.start_new_thread(test,(`count`,))
>             count = count + 1;
>     except:
>         print "Total Threads:", count
>         print "Exiting Main Thread:", thread.get_ident()
>         raise
> 
> def test(input=None):
>     print "count:", thread.get_ident(), input
>     while 1: #keep thread alive until it breaks
>         pass
> 
> if __name__ == "__main__":
>     main()
> 
> #####
> 
> Results:
> 
> 1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
>    Max Threads = 1024
> 
> 2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
>    Max Threads = 1024
> 
> 3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
>    Max Threads = 256
> 
> 4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
>    Max Threads = 512
> 
> Note:
>    For all setup, SuSE Linux threads-max=14336 and max_map_count=65536
> 
> Questions:
> 1. How to determine the number of threads? Is it something
> configurable?
> 2. Why do the results above differ in output?
> 3. How to increase the maximum number of threads per process?
> 

I'm not sure if this is configurable but I was testing out something 
similar today (on fedora core 1). I discovered that a structure such as 
the following works with any number of thread (I tried it with 2000 
threads just now). The main thing I did was introducing a 
'time.sleep(0.01)' in the loop that waits for the worker threads to join 
the main thread, and adding a try..except clause around where you start 
the threads. Basically an exception is never thrown because I run 10 
threads at one go, and then sleep for a fraction of a second (it's 
usually a good idea to sleep/yield in busy loops). I think the main 
thing here is the fact that it takes 10 threads at a time and then gives 
it a pause.

Please try this out and let me know what you find.

class A:
     cache = {}
     counter = 0
     def local(self):
         A.cache[A.counter*2] = A.counter
         A.counter += 1

     def print_it(self):
         print A.cache, len(A.cache), A.counter


def test():
     count = 2000 # I'm pretty sure a larger number should work too

     a = A()
     threadList = []
     for i in xrange(count):
         try:
             t = threading.Thread(target=a.local, args=())
             threadList.append(t)
             t.start()

             # Adding the following if statement allowed running more 
than 256 threads in one go
             # Previously it would run only 256 threads if this clause 
wasn't there.
             if i % 10 == 0 and i != 0:
                 # setting sys.setcheckinterval to 1 didn't help. Let's try
                 # voluntarily giving up some cycles -- this should give
                 # other threads a chance to run
                 time.sleep(0.01)  # this works with time.sleep(0.005) 
as well
             # end if
         except:
             continue
         # end try
     # end for

     # Wait for threads to join the main thread
     while threading.activeCount()>1:
         time.sleep(1)

     a.print_it()

test()


> 
> Thanks and hope to hear soon.
> Ronan

Steve