What's the cost of using hundreds of threads?

[Aahz]

In article <d047h2$b5l$1 at julia.coi.pw.edu.pl>,
=?ISO-8859-2?Q?Przemys=B3aw_R=F3=BFycki?=
  <P.Rozycki at elka.pw.edu.pl> wrote:
>
>I don't spawn them because of computional reasons, but due to the fact
>that it makes my code much more simpler. I use built-in tcp features
>to achieve loadbalancing - every flow (directed through pipe) has it's
>own dedicated threads - separate for down- and upload. For every 'main
>connection' these threads share send and receive buffer. If any of
>pipes is congested the corresponding threads block on their send / recv
>functions - without affecting independence of data flows.
>
>Using threads gives me VERY simple code. To achieve this with poll /
>select would be much more difficult. And to guarantee concurrency and
>maximal throughput for all of pipes I would probably have to mirror
>code from linux TCP stack (I mean window shifting, data acknowlegement,
>retransmission queues). Or perhaps I exaggerate.

Maybe it would help if you explained what these "pipes" do.  Based on
what you've said so far, what I'd do in your situation is create one
thread per pipe and one thread per connection, then use Queue to move
data between threads.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code -- 
not in reams of trivial code that bores the reader to death."  --GvR