[scikit-learn] Implement Catalyst-SVRG optimizer for linear models?

[Krishna Pillutla]

Hello all,

*TL;DR*: I'd like to implement Catalyst-SVRG
<https://arxiv.org/pdf/1712.05654.pdf>, an accelerated optimization
algorithm for sklearn (or scikit-learn-contrib/lightning, if it is more
appropriate). Any feedback?

*Long version*:
I've been playing around with Catalyst-SVRG
<https://arxiv.org/pdf/1712.05654.pdf>, an accelerated stochastic variance
reduced optimization algorithm for my research. I've found in my experience
and in the experiments section of the attached paper that this algorithm
does lead to faster optimization than vanilla (un-accerelated) SVRG
<https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf>,
which itself is much faster than SGD and on roughly the same footing as
SAG/SAGA. Moreover, the per-iteration computational complexity of this
algorithm practically matches that of SVRG.

I was wondering whether it would be beneficial to the community if I
implemented this algorithm in sklearn/linear_model or perhaps
in scikit-learn-contrib/lightning. I would love to hear your thoughts on
this.

Cheers,
Krishna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20180222/285689e5/attachment.html>