Maximum number of threads

[Andrew MacIntyre]

On Wed, 29 Sep 2004, Kirby Angell wrote:

> No, your babbling has gotten me much closer.  I'm not sure what changed
> between Python 2.2 and Python 2.3, but the problem is the stack size
> allocated to each thread.  Apparently with glibc on RH9 something along
> the order of 8 MB of virtual memory is allocated for the stack of each
> thread.  256 threads takes us to the virtual memory limit of the process
> itself. Glibc on FC2 must be using a different default stack size, or
> the virtual memory limit is higher; I've only recently started running
> FC2 on our test servers so I'm not sure which it is.
>
> The only way to change the stack size is to define THREAD_STACK_SIZE in
> the thread_pthread.h header.  There is an ifdef for the Mac that sets
> this to 64k.  Just as a test I forced THREAD_STACK_SIZE to that value
> and recompiled Python.  I can now create 16k threads although I'm not
> sure yet whether our application will run; some tuning is probably in
> order.  I'm going to get the 2.2 source code and see what is different
> between the two and maybe get an idea of what a reasonable stack size is.

ISTR that the default stacksize for the LinuxThreads threading
implementation (prior to NPTL) was 1MB.

It is also worth noting that more recent versions of gcc (3.x) are
generating larger stack frames than older versions; something that has
affected building the SRE module until it was made non-recursive for
Python 2.4.  This has probably encouraged the bump in the default stack
size.

-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia