multiprocessing as batch system

Neal Becker ndbecker2 at gmail.com
Thu Jan 21 07:13:48 EST 2010


I'm using multiprocessing as a poor man's batch system.  It is working OK, 
except that it doesn't seem to do load balancing quite right.

I have a 8-cpu machine.  If I start, using multiprocessing pool, calling map 
with more than 8 elements, it will start 8 processes.  It seems, though, 
that when some of them finish, it does not start others right away.  The 
load average may be a lot less than 8 but still additional processes are not 
active.

Is there a way to use multiprocessing to achieve the effect I want?  (I 
really want a simple batch system, but without complex setup) 




More information about the Python-list mailing list