Tips for load balancing multiple Python apps on dual/quad core processors?

[Steve Holden]

Malcolm Greene wrote:
> I'm looking for tips on how to load balance running multiple Python
> applications in multi-CPU environments. My understanding is that Python
> applications and their threads are limited to a specific CPU.
> 
> Background: I have a Python utility that processes email messages. I
> suspect there's a lot of idle time while this utility waits on a remote
> email server. I would like to run as many simultaneous copies of this
> utility as possible without overwhelming the server these utilities are
> running on. My thought is that I should write a dispatcher that monitors
> CPU load and launches/cancels multiple instances of my utility with
> specific job queues to process. 
> 
> Is there a cross-platform way to monitor CPU load?
> 
> Is there a pre-existing Python module I should be looking at for
> building (subclassing) my dispatcher?
> 
As a data point for you, I wrote a multi-threaded application that used 
100 threads to send about 45,000 emails in less than four hours. The 
single-CPU computer that processed this job wasn't using that much CPU 
(certainly under 50%).

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/