multiprocessing as batch system

[Neal Becker]

Adam Tauno Williams wrote:

> On Thu, 2010-01-21 at 07:13 -0500, Neal Becker wrote:
>> I'm using multiprocessing as a poor man's batch system.  It is working
>> OK, except that it doesn't seem to do load balancing quite right.
>> I have a 8-cpu machine.  If I start, using multiprocessing pool, calling
>> map
>> with more than 8 elements, it will start 8 processes.  It seems, though,
>> that when some of them finish, it does not start others right away.  The
>> load average may be a lot less than 8 but still additional processes are
>> not active.
> 
> If your processes are I/O bound your system will never appear to be
> saturated.  Multiple pending processes and a low load average usually
> indicated waiting on some other resource.
No, 100% compute bound.

My impression was that some processes completed, but new ones were not 
immediately started.