[SciPy-Dev] Extending scipy.optimize.minimize with particle swarm

[Matthias Cuntz]

Dear all, 

Andrea convinced to join the mailing list and send the code for the Shuffle Complex Evolution (SCE) algorithm. The code can be found here:
https://github.com/mcuntz/jams_python/blob/master/jams/sce.py

This code is based on a Fortran program of Qingyun Duan (2004), ported to Python by Stijn Van Hoey (2011, https://github.com/stijnvanhoey/Optimization_SCE). I took it up, debugged and enhanced it, and are maintaining it since 2013. It followed pretty much the development Juliane Mai and I have done on the Fortran version (https://github.com/mcuntz/jams_fortran/blob/master/mo_sce.f90).

Documentation is in numpydoc format. But I have no experience about scipy standards.

The code has, for example, the possibility to restart an optimisation, which is very suitable for expensive model evaluations (several hours in our examples). So the code writes out the restart information by default (can be switched off), which might be a side-effect unappreciated by scipy.

Kind regards
Matthias


>> Hi Matt,
>
> On Sat, 16 May 2020 at 00.01, Matt Newville <newville at cars.uchicago.edu>
> wrote:
>
>> Hi Andrea,
>>
>> On Fri, May 15, 2020 at 10:42 AM Andrea Gavana <andrea.gavana at gmail.com>
>> wrote:
>>
>>> On Fri, 15 May 2020 at 17.21, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Thu, May 14, 2020 at 3:55 AM Andrew Nelson <andyfaff at gmail.com>
>>>> wrote:
>>>>
>>>>> As a global optimizer in the first instance it would probably be added
>>>>> as a separate function (`particle_swarm`), rather than be added as a method
>>>>> to minimize. I'm not familiar with particle swarm, are there various
>>>>> types/flavours of the approach?
>>>>>
>>>>> The general route to adding new global optimizers is:
>>>>>
>>>>
>>>> Thanks for this detailed reply Andrew. I think it's worth adding this to
>>>> the new section of the developer guide once
>>>> https://github.com/scipy/scipy/pull/12069 is merged.
>>>>
>>>> One thing to ask additionally here is: are particle swarm methods in
>>>> scope? It would be good to compare this proposed method to for example
>>>> what's currently available in PyGMO (https://esa.github.io/pygmo/).
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>
>>> My apologies for pitching in, I’m usually just an observer. I’m sure some
>>> of you remember the global optimization benchmarks I once posted - and I
>>> believe they have been updated, adapted and integrated in scipy:
>>>
>>> http://infinity77.net/global_optimization/multidimensional.html
>>>
>>> In the set of optimizers, a particle swarm algorithm was actually there,
>>> and specifically the one from:
>>>
>>> http://www.norg.uminho.pt/aivaz/pswarm/
>>>
>>> And it turned out to be not so bad in the end. My impression is that
>>> there might be scope for a particle swarm algorithm in scipy: I’ve used a
>>> few variants of them in the past where many other optimization procedures
>>> failed. But then I’m just a lurker so please take my opinion with a grain
>>> of salt :-).
>>>
>>> Andrea.
>>>
>>
>> Thanks - I think your benchmarks are very informative and useful.  FWIW,
>> we have implemented the `ampgo` solver to lmfit (
>> https://lmfit.github.io/lmfit-py/fitting.html#lmfit.minimizer.Minimizer.ampgo),
>> mainly based on your results.
>>
>
>Thank you, I was actually surprised one day (a while ago) while doing a
>google search for AMPGO to find that it had been integrated into lmfit. I’m
>glad it could be helpful. I always thought it could be improved by
>implementing the gradient of the Tunnelling function (if the user has
>analytical gradient of his/her function) but my math skills were too weak
>to manage that...
>
>
>> It looks to me like the next most useful to add would be `sce`.   That is,
>> it appears to be roughly "second best" and pretty good at being disjoint to
>> `ampgo`, `basinhopping`, and `diffev`.   Do you have code for this `sce`
>> method that you could share?  I'm afraid I know absolutely nothing about
>> this method.   I must admit that it was only from your work that I learned
>> about `ampgo` too.
>>
>> Based on your results, it seems to me that adding `ampgo` and `sce` to
>> scipy.optimize would be higher priorities than adding the particle smarm
>> optimizer.   Sure, every method has value, but using your benchmarks might
>> be a good way to prioritize which methods to add.
>>
>
>I do have the code for SCE but I’m not the original author: I was contacted
>about 6 years ago via private email by Matthias Cuntz and he provided me
>the code I tested: I can of course try and get in touch with him and see if
>he’s willing to share it with the community - assuming there is interest
>for this algorithm and that he (or someone else) is willing to bring it to
>the standards of SciPy code (or other libraries in the Python Scientific
>stack).
>
>
>> Cheers, and thanks again,
>>
>> --Matt
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev