Simple thread pools

[Steve Holden]

Josiah Carlson wrote:

> Jacob Friis <lists at debpro.webcom.dk> wrote:
> 
>>I have built a script inspired by a post on Speno's Pythonic Avocado:
>>http://www.pycs.net/users/0000231/weblog/2004/01/04.html#P10
>>
>>I'm setting NUM_FEEDERS to 1000.
>>Is that crazy?
> 
> 
> Not crazy, but foolish.  Thread scheduling in Python reduces performance
> beyond a few dozen threads.  If you are doing system calls (socket.recv,
> file.read, etc.), your performance will be poor.
> 
Is this speculative, or do you have some hard evidence to support it? I 
recently rewrote a billing program that delivers statements by email. 
The number of threads it uses is a parameter to the program, and we are 
currently running at 200 with every evidence of satisfaction - this 
month's live run sent something over 10,000 emails an hour.
> 
>>Are there a better solution?
> 
> 
> Fewer threads.  Try running at 10-30.  If you are finding that you
> aren't able to handle the load with those threads, then your
> processor/disk/etc isn't fast enough to handle the load.
> 
I'm tempted to say "rubbish", but that would be rude, so instead I'll 
just ask for some evidence :-). Don't forget that in network-based tasks 
the time spent waiting for connection turnarounds can dominate the 
elapsed time for execution - did you perhaps overlook that?

regards
  Steve
-- 
http://www.holdenweb.com
http://pydish.holdenweb.com
Holden Web LLC +1 800 494 3119