[SciPy-User] lmfit: confidence intervals issue

[Matt Newville]

Hi Antonino,

On Wed, Jun 18, 2014 at 6:12 PM, Antonino Ingargiola <tritemio at gmail.com>
wrote:
>
> Hi,
>
> I'm not sure this is the right list for this kind of issues.


It's OK with me.  Using github issues would be reasonable too.

>
>
> I'm using lmfit to solve a non-linear least square problem to fit a
"noisy" histogram to a model that is a single exponential decay convoluted
with a "sharp" peak and a baseline noise floor.
>
> The problem is that unless I set constrains for the 'offset' parameter
the confidence interval calculation fails in a strange way in scipy brentq
function.
>
> You can see the full example notebook and the error message in this
ipython notebook:
>
>
http://nbviewer.ipython.org/urls/gist.githubusercontent.com/tritemio/901c2eb2a43775e81844/raw/755eede8f170cc33b96316601286ae534e901c49/lmfit%20issue
>
> What can I do to fix the problem? If this is the normal behavior, a more
informative error message would be helpful.

I haven't tried your example, but it's possible that the development
version on github.com fixes this -- there was a similar problem with
 conf_interval() failing when calling brentq() that was fixed relatively
recently (slow devel cycle),  but I'm not certain that the fix there will
work here.

I would guess that the essential problem is related to the fact that your
also getting NaNs from the covariance matrix.  Again, without actually
trying your problem, I believe that this is probably related to your use of

     pos_range = (x - offset) >= 0

to select the data range for which y!=0.   This could make the fit
insensitive to small changes (ie, well below 1) to offset, making it hard
to compute the partial derivative for it.  This could, in turn, make the
covariance matrix hard to get (which is what you are seeing).    Without
finite, numerical values for stderr from the covariance matrix, it's hard
for conf_interval() to make good guesses about what range of values to
search around.

In short, it's a challenge to  have variables that are used essentially as
integers.  Still, even if it doesn't work well, it shouldn't be failing
this way.

My advice would be to try the devel version of lmfit.  If it's still
failing, try to construct a smaller failing example (or make something
complete that can be run as a non-IPython script), and start an Issue.

Cheers,

--Matt Newville
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140618/53597444/attachment.html>