What's the cost of using hundreds of threads?

[Aahz]

In article <d0295h$hfc$1 at julia.coi.pw.edu.pl>,
=?ISO-8859-2?Q?Przemys=B3aw_R=F3=BFycki?=
  <P.Rozycki at elka.pw.edu.pl> wrote:
>
>I have written some code, which creates many threads for each connection 
>('main connection'). The purpose of this code is to balance the load 
>between several connections ('pipes'). The number of spawned threads 
>depends on how many pipes I create (= 2*n+2, where n is the number of 
>pipes).
>
>For good results I'll presumably share main connection's load between 10 
>pipes - therefore 22 threads will be spawned. Now if about 50 
>connections are forwarded the number of threads rises to thousand of 
>threads (or several thousands if even more connections are established).

I'm a bit confused by your math.  Fifty connections should be 102
threads, which is quite reasonable.

>My questions are:
>- What is the cost (in memory / CPU usage) of creating such amounts of 
>threads?
>- Is there any 'upper boundary' that limits the number of threads? (is 
>it python / OS related)
>- Is that the sign of 'clumsy programming' - i.e. Is creating so many 
>threads a bad habit? (I must say that it simplified the solution of my 
>problem very much).
>
>Limiting the number of threads is possible, but would affect the 
>independence of data flows. (ok I admit - creating tricky algorithm 
>could perhaps gurantee concurrency without spawning so many threads - 
>but it's the simplest solution to this problem :) ).

My experience with lots of threads dates back to Python 1.5.2, but I
rarely saw much improvement with more than a hundred threads, even for
heavily I/O-bound applications on a multi-CPU system.  However, if your
focus is algorithmic complexity, you should be able to handle a couple of
thousand threads easily enough.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code -- 
not in reams of trivial code that bores the reader to death."  --GvR