[SciPy-Dev] Extending scipy.optimize.minimize with particle swarm

[Andrea Gavana]

Hi Andrew,

On Sat, 16 May 2020 at 01.21, Andrew Nelson <andyfaff at gmail.com> wrote:

> Andrea's test function collection is super nice, and is able to produce
> good evidence for assessment in this area. I do have one minor disagreement
> with the way the benchmarks on
> http://infinity77.net/global_optimization/index.html were run, the
> maximum number of function evaluations is limited to 2 000.
>

I will have to thank you for the immense pain you went through in
painstakingly checking the benchmarks and fixing all the errors I put in
the test functions myself and bringing the test suite up to scipy
standards. But I’m happy they’re there now :-) .
One of the problem in my field of work (and I’m pretty sure I’m many other
disciplines) is that evaluating the objective function is often a very time
consuming process of phenomenal complexity - hours of simulation time on
heavily parallellized code running on supercomputers. So 2,000 function
evaluations as a limit was actually quite generous. I do recognize though
that many real life problems do not exhibit the same irregularities of the
test functions - if your model resembles anything like the Griewank
function or the CrossInTray function then there must be something fishy in
the model itself...


> In any case, it's clear that implementations matter. For example,
> http://infinity77.net/global_optimization/multidimensional.html# would
> indicate that differential evolution isn't very good against the Ackley
> function, with zero successes. However, running the scipy implementation of
> differential-evolution indicates that it's actually quite good,
> https://gist.github.com/andyfaff/24c96a3d5dbc7b0272b2. The latter gist
> would approximate the output from running the optimize benchmarks.
>
> (Note that the gist is quite old, and the `anneal` column is from the old
> anneal function that was removed)
>

I definitely agree that implementation matters: I was actually surprised
and disappointed by the original DE code, and I’m happy that the code in
scipy manages to show much better results.

Andrea.



>
>
> On Sat, 16 May 2020 at 08:02, Matt Newville <newville at cars.uchicago.edu>
> wrote:
>
>> Hi Andrea,
>>
>> On Fri, May 15, 2020 at 10:42 AM Andrea Gavana <andrea.gavana at gmail.com>
>> wrote:
>>
>>> On Fri, 15 May 2020 at 17.21, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Thu, May 14, 2020 at 3:55 AM Andrew Nelson <andyfaff at gmail.com>
>>>> wrote:
>>>>
>>>>> As a global optimizer in the first instance it would probably be added
>>>>> as a separate function (`particle_swarm`), rather than be added as a method
>>>>> to minimize. I'm not familiar with particle swarm, are there various
>>>>> types/flavours of the approach?
>>>>>
>>>>> The general route to adding new global optimizers is:
>>>>>
>>>>
>>>> Thanks for this detailed reply Andrew. I think it's worth adding this
>>>> to the new section of the developer guide once
>>>> https://github.com/scipy/scipy/pull/12069 is merged.
>>>>
>>>> One thing to ask additionally here is: are particle swarm methods in
>>>> scope? It would be good to compare this proposed method to for example
>>>> what's currently available in PyGMO (https://esa.github.io/pygmo/).
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>
>>> My apologies for pitching in, I’m usually just an observer. I’m sure
>>> some of you remember the global optimization benchmarks I once posted - and
>>> I believe they have been updated, adapted and integrated in scipy:
>>>
>>> http://infinity77.net/global_optimization/multidimensional.html
>>>
>>> In the set of optimizers, a particle swarm algorithm was actually there,
>>> and specifically the one from:
>>>
>>> http://www.norg.uminho.pt/aivaz/pswarm/
>>>
>>> And it turned out to be not so bad in the end. My impression is that
>>> there might be scope for a particle swarm algorithm in scipy: I’ve used a
>>> few variants of them in the past where many other optimization procedures
>>> failed. But then I’m just a lurker so please take my opinion with a grain
>>> of salt :-).
>>>
>>> Andrea.
>>>
>>
>> Thanks - I think your benchmarks are very informative and useful.  FWIW,
>> we have implemented the `ampgo` solver to lmfit (
>> https://lmfit.github.io/lmfit-py/fitting.html#lmfit.minimizer.Minimizer.ampgo),
>> mainly based on your results.
>>
>> It looks to me like the next most useful to add would be `sce`.   That
>> is, it appears to be roughly "second best" and pretty good at being
>> disjoint to `ampgo`, `basinhopping`, and `diffev`.   Do you have code for
>> this `sce` method that you could share?  I'm afraid I know absolutely
>> nothing about this method.   I must admit that it was only from your work
>> that I learned about `ampgo` too.
>>
>> Based on your results, it seems to me that adding `ampgo` and `sce` to
>> scipy.optimize would be higher priorities than adding the particle smarm
>> optimizer.   Sure, every method has value, but using your benchmarks might
>> be a good way to prioritize which methods to add.
>>
>> Cheers, and thanks again,
>>
>> --Matt
>> _______________________________________________
>> SciPy-Dev mailing list
>> SciPy-Dev at python.org
>> https://mail.python.org/mailman/listinfo/scipy-dev
>>
>
>
> --
> _____________________________________
> Dr. Andrew Nelson
>
>
> _____________________________________
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20200516/30d52337/attachment.html>