[SciPy-Dev] GSoC SciPy Optimization Ideas

[Andrew Nelson]

Dear Mazen,
I don't hink that the proposals you outlined are clear enough at the
moment, and would benefit from addressing the following points.

- the penalty method is the most interesting part of the proposal. I think
a bit more expansion on this point would be useful. For example, how easy
would it be to turn any constrained problem into an objective function that
could be used with any unconstrained minimizer? Could that respecification
work with existing interfaces for expressing constraints? Could the
existing minimizer interface be left untouched, and function wrappers or
classes written to convert those constrained systems into a function given
to unmodified existing code?

- Vectorisation in optimize sounds like a nice idea, but may be very
difficult to implement in practice. From your document it seems that you're
mainly proposing to vectorise derivative calculation using finite
differences. The finite difference routine `approx_derivative` in _numdiff
would have to be rewritten to perform in a vectorised manner, across
different dimensions (R^m -> R->n), with a vectorised objective function.
How would the stack trace from the initial `minimize` call through to
approx_derivative have to change? (this is definitely way more than a weeks
work). If you had profiling to demonstrate that vectorisation would provide
a significant performance gain that would be great.

- it's unclear what the normalisation (section 5) refers to? Is this an
automated way of selecting the correct value for a Lagrangian multiplier?

- is the smart initialisation (section 7.2) a way of selecting the correct
value for a Lagrangian multiplier used for the constrained->unconstrained
transform? Or is it a variation on `brute + polish`? If the latter then how
would this be any better than shgo/dual_annealing/differential_evolution?

- improvement of documentation is always welcomed, but there's not a huge
explanation of the specific target areas for improvement.

regards,
Andrew.

On Mon, 12 Apr 2021 at 19:06, Mazen Sayed <sayedmazen70 at gmail.com> wrote:

> Hello,
>
> I have changed the normalizer to be a user specified argument, also I have
> changed the initializer so the user can enter multiple data points, and the
> initializer will take the best initial point (that minimize the
> unconstrained problem), or the user can let the initializer guess the
> initial point by using random search, grid search or global optimizer.
>
> Any modification before submission?
>
> Thanks
>
>
> https://drive.google.com/file/d/1c9JOJgcq_Ss761rt9SOxnmEtuczuLzAb/view?usp=sharing
>
>
>
> On Sun, Apr 11, 2021 at 7:26 PM Mazen Sayed <sayedmazen70 at gmail.com>
> wrote:
>
>>
>> This sounds really good and more reasonable.
>>
>> Thanks for your help.
>>
>>
>>
>> On Sun, Apr 11, 2021 at 7:08 PM Pamphile Roy <roy.pamphile at gmail.com>
>> wrote:
>>
>>> I tend to agree with Daniel here, randomly choosing 50 points in a
>>> high-dimensional optimization space is not going to give any advantage. And
>>> why 50?
>>>
>>> The initialization part is one of the most important (and difficult to
>>> get right) part of any optimization algorithm, but this is mostly true for
>>> global ones: differential evolution, SHGO, Dual Annealing they’re all have
>>> their own way. Some of these and many others (especially local algorithms)
>>> rely on the user to explicitly pass an initial guess and take it from there.
>>>
>>>
>>> Agreed here, the random sampling must not be totally random in order to
>>> cover the parameter space in the most efficient way.
>>> The global optimizers all use QMC methods (scipy.stats.qmc) so here you
>>> can just rely on these too.
>>>
>>> Cheers,
>>> Pamphile
>>>
>>> _______________________________________________
>>> SciPy-Dev mailing list
>>> SciPy-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/scipy-dev
>>>
>> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>


-- 
_____________________________________
Dr. Andrew Nelson


_____________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scipy-dev/attachments/20210414/ec5aa08f/attachment.html>