[SciPy-User] Distributed computing: running embarrassingly parallel (python/c++) codes over a cluster

[Pauli Virtanen]

Mon, 09 Nov 2009 23:41:29 +0530, Rohit Garg wrote:
[clip: embarassingly parallel problems]

With multiprocessing, using Pool.imap_unordered to apply a computation 
function to a list of parameter sets is one good alternative. (IIRC, it 
balances load between subprocesses &c automatically.) Multiprocessing can 
however work on only one node at a time.

With mpi4py, it's probably best to write a simple master-slave 
architecture.

-- 
Pauli Virtanen