[SciPy-Dev] Large Scale Optimization algorithm implementation

[Stephan Hoyer]

These algorithms look specialized to problems where the optimization
objective can be written as an average over many examples.

This is not necessarily the case for the functions optimized by
scipy.optimize.minimize -- we don't have any equivalent of stochastic
gradient descent. This might be more suitable for libraries that specialize
in these sort of optimization problems, like TensorFlow or pytorch.

On Fri, Sep 14, 2018 at 3:42 PM Touqir Sajed <touqir at ualberta.ca> wrote:

> Hi,
>
> So, the most efficient optimization algorithm implemented in Scipy is
> LBFGS-B in terms of memory consumption and runtime speed. It would be good
> to see algorithms suited for large scale optimization in scipy for the
> scenarios when Gradient computation is quite expensive and/or Number of
> variables is huge. SVRG and (SVRG+LBFGS) can both attain much better
> performance than LBFGS-B for this scenario.
> The links:
> (SVRG) :
> https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf
> <https://ml-trckr.com/link/https%3A%2F%2Fpapers.nips.cc%2Fpaper%2F4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf/dzjpYvcNc3EU6zyPRAsq>
> (SVRG + LBFGS) : http://opt-ml.org/papers/OPT2015_paper_41.pdf
> <https://ml-trckr.com/link/http%3A%2F%2Fopt-ml.org%2Fpapers%2FOPT2015_paper_41.pdf/dzjpYvcNc3EU6zyPRAsq>
>
> What do you think?
>
> Cheers,
> Touqir
>
> --
> Computing Science Master's student at University of Alberta, Canada,
> specializing in Machine Learning. Website :
> https://ca.linkedin.com/in/touqir-sajed-6a95b1126
> <https://ml-trckr.com/link/https%3A%2F%2Fca.linkedin.com%2Fin%2Ftouqir-sajed-6a95b1126/dzjpYvcNc3EU6zyPRAsq>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20180914/4c4a7032/attachment.html>