multiprocessing as batch system

[Adam Tauno Williams]

On Thu, 2010-01-21 at 07:13 -0500, Neal Becker wrote:
> I'm using multiprocessing as a poor man's batch system.  It is working OK, 
> except that it doesn't seem to do load balancing quite right.
> I have a 8-cpu machine.  If I start, using multiprocessing pool, calling map 
> with more than 8 elements, it will start 8 processes.  It seems, though, 
> that when some of them finish, it does not start others right away.  The 
> load average may be a lot less than 8 but still additional processes are not 
> active.

If your processes are I/O bound your system will never appear to be
saturated.  Multiple pending processes and a low load average usually
indicated waiting on some other resource.

> Is there a way to use multiprocessing to achieve the effect I want?  (I 
> really want a simple batch system, but without complex setup) 

Well, honestly, that isn't possible (unless you just choose to ignore
all the things that can go wrong: crashed worked, hung workers, etc...)

-- 
OpenGroupware developer: awilliam at whitemice.org
<http://whitemiceconsulting.blogspot.com/>
OpenGroupare & Cyrus IMAPd documenation @
<http://docs.opengroupware.org/Members/whitemice/wmogag/file_view>