[Numpy-discussion] Inversion of near singular matrices.

[Algis Kabaila]

On Sunday 30 January 2011 09:10:30 Sturla Molden wrote:
> Den 29.01.2011 12:40, skrev Algis Kabaila:
> > So my question is: how can one reliably detect singularity
> > (or near singularity) and raise an exception?
> 
> Use an SVD, examine the singular values. 
I gather that SVD is the Singular Value Decomposition, but I 
have no idea how to perform such decomposition. Would you care 
to refer me to some simple source material?  I have been advised 
to watch the condition numbers. No doubt, SVD and condition 
numbers are related.  The references about condition numbers are 
very interesting and I intend to follow them in the first 
instance.

> In statistics we sometimes see ill-conditioning of covariance
> matrices. Another way to deal with multicollinearity besides
> SVD/PCA is regularisation. Simply adding a small bias k*I to
> the diagonal might fix the problem (cf. ridge regression).
> In the Levenberg-Marquardt algorithm used to fit non-linear
> least squares models (cf.
> scipy.optimize.leastsq), the bias k to the diagonal of the
> Jacobian is changed adaptively. One might also know in
> advance if a covariance matrix could be ill-conditioned (the
> number of samples is small compared to the number of
> dimensions) or singular (less data than parameters). That
> is, sometimes we don't even need to look at the matrix to
> give the correct diagnosis. Another widely used strategy is
> to use Cholesky factorization on covariance matrices. It is
> always stable unless there is a singularity, for which it
> will fail (NumPy will raise a LinAlgError exception).
> Cholesky is therefore safer to use for inverting covariance
> matrices than LU (as well as faster). If Cholesky fails one
> might fallback to SVD or regularisation to correct the
> problem.
>
> Sturla
>
My knowledge of statistics is rather limited, though our son Dr. 
Paul Kabaila is a specialist in that area.  My interests lie in 
the area of Analysis of Engineering Structures - it saves my 
brain from falling to a permafrost like sleep   :)

Thank you for your reply - greatly appreciated.

Al.

PS: Paul, I thought there is a minuscule chance that this is of 
some interest to you.
Tete.
-- 
Algis
http://akabaila.pcug.org.au/StructuralAnalysis.pdf