[SciPy-User] fmin_l_bfgs_b issues with small numbers

[Gilles Rochefort]

Hi,

My very first question is why are you using a multivariate optimization
such as lbfgs for your problem ?

>From what I understood your problem is just optimizing a function with only
one scalar unknown x.

L-BFGS is designed for large size problems, when you cannot explicitly
compute the hessian.
The L stands for limited memory ... where the hessian is approximated by
means of a suitable decomposition.

Also, here, you are asking lbfgs to approximate your first and second
derivative.

By the way, I notice that the function you give is convex between [-1;1],
so you should use fminbound to find the solution :

In [1]: from scipy.optimize import *

In [2]: fminbound(optim_par, -1.0, 1.0,disp=3)

 Func-count     x          f(x)          Procedure
    1      -0.236068  1.12379e-07        initial
    2       0.236068  1.09953e-07        golden
    3       0.527864  4.56894e-07        golden
    4     0.00164419  4.45293e-09        parabolic
    5      0.0013422  4.44086e-09        parabolic
    6     -0.0099419   4.2406e-09        parabolic
    7     -0.0963144  1.94664e-08        golden
    8    -0.00892899  4.23856e-09        parabolic
    9    -0.00891099  4.23856e-09        parabolic
   10    -0.00891432  4.23856e-09        parabolic
   11   * -0.00891765*  4.23856e-09        parabolic


In [3]: optim_par( -0.00891296 )
Out[3]: *4.2385592303771674e-09*

In [4]: optim_par( -0.0296399132248 )
Out[4]:* 5.0744070563855195e-09*


Obviously, you should also use another algorithm which would be really more
efficient if you provide the first and second derivative of your function.

Regards,
Gilles.


2011/12/16 Daniel Arribas-Bel <darribas at asu.edu>

> Hi all!
>
> [I hope this hasn't been asked before on the list, I haven't found
> anything similar at least.]
>
> I have a simple optimization problem that involves two very small arrays
> of dimension (2, 2) and (2, 1):
>
> aa = np.array([[ 0.00030763, -0.00011521], \
>         [ 0.00093007, -0.00015189]])
> a = np.array([[  2.54854751e-05], [ -3.93219333e-05]])
>
>
> The function to optimize is:
>
> optim_par = lambda par: np.sum((np.dot(aa, np.array([par, par**2])) -
> a)**2)
>
> Now, if I just run optimize.fmin_l_bfgs_b with the default parameters:
>
>> start = [0.0]
>> bounds=[(-1.0,1.0)]
>> ll = op.fmin_l_bfgs_b(optim_par, start, approx_grad = True, bounds =
>> bounds)
>>
>
> I get a parameter of exactly 0.0, same as the starting value.
>
> If instead I either modify the parameters 'pgtol' and 'factr':
>
> l = op.fmin_l_bfgs_b(optim_par, start, approx_grad = True, bounds =
>> bounds, pgtol = 1e-50, factr = 10.0)
>>
>
> or simply re-scale the arrays multiplying them by 10,000 and run it as
> initially, the parameter I get is in either case -0.0296399132248. The
> attached script runs the three options.
>
> A couple of questions about this:
>
>    - Is this caused because of the way the optimizer handles relatively
>    small numbers or am I missing something else?
>    - I want this to run inside an app that will take different datasets
>    even though the arrays 'a' and 'aa' will always be of that shape. Which of
>    the two solutions would you recommend as more stable, robust and fast? Why?
>
> Any other comment/suggestion is of course most welcome. Thank you very
> much in advance,
>
> ]d[
>
> --
> ============================================================
> Daniel Arribas-Bel, PhD.
> Url: darribas.org
> Mail: darribas at asu.edu
>
> GeoDa Center for Geospatial Analysis and Computation (geodacenter.asu.edu)
> Arizona State University (USA)
> ============================================================
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20111216/3b7cfa6f/attachment.html>