Any suggestion to start more threads at the same time?

[Christian Heimes]

Am 28.07.2011 23:07, schrieb smith jack:
> I start many threads in order to make the work done, when the
> concurrent number is set to 300, all thing just works fine, but when
> the number is set to 350 or higher, error just comes out? what's wrong
> ? the error info is just as follows:   failed to start .
> 
> I am confused, does this have something to do with the operating
> system, i am now using Linux, any suggestion to make the system to
> support more python threads?

300 threads is an insanely high amount of threads for almost every
environment. For CPython threads won't speed up your program if it's CPU
bound and spends most of its time in Python space (no C code that
releases the GIL). For IO bound code you are going to get much better
performance with some kind of async event loop (select, poll, epoll).
Twisted's event reactor may help here, too.

With that many threads any program (not limited to Python) will spend a
considerable amount of CPU resources to manage the threads. Every thread
also consumes a fair amount of memory since every thread needs its own C
stack. On most systems the C stack for every thread is about 8 MB. There
are also some management things that need memory.

You can reduce the stack size *before* you start any threads:

import sys
import thread
import resource

sys.setrecursionlimit(500) # default is 1000
stacksize = 4 * 1024 * 1024 # default is 8 MB on most systems
thread.stack_size(stacksize)
resource.setrlimit(resource.RLIMIT_STACK, (stacksize, -1))

That should give you some more spare memory.

Christian