[SciPy-User] Fitting procedure to take advantage of cluster

[Christoph Deil]

On Jun 29, 2011, at 6:54 PM, J. David Lee wrote:

> I'm attempting to perform a fit of a model function's output to some 
> measured data. The model has around 12 parameters, and takes tens of 
> minutes to run. I have access to a cluster with several thousand 
> processors that can run the simulations in parallel, so I'm wondering if 
> there are any algorithms out there that I can use to leverage this 
> computing power to efficiently solve my problem - that is, besides grid 
> searches or Monte-Carlo methods.

Hi David,

there are two somewhat unrelated questions:
1) What optimizer can minimize your cost function (a.k.a. fit statistic) with the least evaluations?
2) How can you parallelize the computation, taking advantage of 1000s of cores?

We can't give good advice on these questions without knowing more about your data, model and cost function.
In principle I think any optimizer can be parallelized.

There are many optimizers, you can find a short description of three common ones (Levenberg-Marquardt, Simplex, Monte-Carlo) here:
http://cxc.harvard.edu/sherpa/methods/index.html
I would recommend trying Levenberg-Marquardt first because it is very fast, but you need reasonable starting parameters and if your cost function is complicated you can get stuck in a local minimum.

Concerning the parallelization, since you mentioned that one evaluation of the cost function takes 10 minutes, if you can, split the data into small independent chunks, distribute them to each node (once) with the current parameter set (for each iteration), and report the fit statistic for this chunk back to the master process, which then simply sums to get the total statistic. You can find an example with a few lines of python code using Sherpa for fitting and MPI for parallelization here (I haven't tried this example myself):
http://cxc.harvard.edu/ciao/workshop/feb10/talks/aldcroft.pdf (slides 13 to 15)
I've heard very good things about pyzmq (http://www.zeromq.org/bindings:python), which is certainly easier to learn than MPI if you haven't used either before.
Of course such trivial "data-parallelization" is only possible if you can split your dataset into *independent* chunks, i.e. there are no long-reaching correlations between data points.

I hope that helps a bit, in any case you probably need to implement the parallelization yourself and experiment a bit which optimizer works well for your problem.

Christoph