Recommended number of threads? (in CPython)

mk mrkafk at gmail.com
Thu Oct 29 11:56:42 EDT 2009


Hello everyone,

I wrote run-of-the-mill program for concurrent execution of ssh command 
over a large number of hosts. (someone may ask why reinvent the wheel 
when there's pssh and shmux around -- I'm not happy with working details 
and lack of some options in either program)

The program has a working queue of threads so that no more than 
maxthreads number are created and working at particular time.

But this begs the question: what is the recommended number of threads 
working concurrently? If it's dependent on task, the task is: open ssh 
connection, execute command (then the main thread loops over the queue 
and if the thread is finished, it closes ssh connection and does .join() 
on the thread)

I found that when using more than several hundred threads causes weird 
exceptions to be thrown *sometimes* (rarely actually, but it happens 
from time to time). Although that might be dependent on modules used in 
threads (I'm using paramiko, which is claimed to be thread safe).





More information about the Python-list mailing list