Thread limit and thread stack size

[blair.bethwaite at gmail.com]

Hi All,

We've been debugging a distributed app that uses a database server to
accept connections and then queries from remote agents. Once a
connection is established it is passed off to a thread, along with a
local db connection from a pool, to handle the IO. We get to a point
that seems to be a critical connection load where the number of active
threads shoots up until the database server can no longer create a new
thread after socket accept. That point is about 302 threads.

In investigating the issue we found some previous posts regarding
thread limits and modified a code snippet to iteratively try different
thread stack sizes [1], as it stands below the script will create as
many threads as possible for each thread stack size between 128KiB and
20MiB in 128KiB steps.

What I don't understand and I'm hoping somebody can enlighten me
about, is what the practical effect of altering this setting is on the
memory allocation of the interpreter. Running this on different
machines with different virtual memory sizes I expected to see
different values but got the same results on a box with 2x the memory
of the first. I also noted that python is clearly not trying to
allocate stack_size memory for each thread because on a box with only
1.5G vmem I got:
--
150 threads before:
can't start new thread
Setting thread.stack_size(20352K)
--
and 150 * 20352KiB = 2.9GiB, which is larger than the total virtual
memory - so why was I even allowed to create this many threads in the
first place? I guess I'm missing something about the details of
underlying pthread implementation... what gives?

~Blair

[1] threadTest.py:
import thread, time

class threadTest(object):
    def __init__(self):
        self.stop = False

    def tester(self):
        while not self.stop:
            time.sleep(1)

    def dotest(self):
        count = 0
        try:
            while 1:
                thread.start_new_thread(self.tester,())
                count = count + 1
        except Exception,e:
            self.stop = True
            time.sleep(2)
        return (count,e)

if __name__ == "__main__":
    page_size = 4096  # 4 KiB
    step = page_size * 8 * 4  # 128 KiB
    min_stack = 32768 * 4  # 128 KiB
    max_stack = 32768 * 640  # 20 MiB
    count = -1
    for stack_size in xrange(min_stack,max_stack,step):
        print "Setting thread.stack_size(%sK)" % (stack_size/1024)
        try:
            thread.stack_size(stack_size)
        except Exception,e:
            print "Couldn't set stack size: %s" % e
            continue
        try:
            tester = threadTest()
            (count,e) = tester.dotest()
            del tester
            print "%s threads before:\n%s" % (count,e)
        except Exception,e:
            print "Thread count = ",count
            print e