[SciPy-User] scipy.stats logpdf gradients

[josef.pktd at gmail.com]

On Sat, Feb 26, 2011 at 11:50 PM, John Salvatier
<jsalvati at u.washington.edu> wrote:
> Hello,
>
> A while back there was a discussion that ultimately got the logpdf,
> logcdf etc included in scipy.stats
> (http://mail.scipy.org/pipermail/scipy-user/2010-May/025444.html).
> There was some mention of possibly also doing the the gradients of the
> logpdf. the gradients of the logpdf are useful in fitting algorithms.
>
> I am curious whether any work has been done on that front. I would be
> interested in contributing to such work. Is has anyone worked on this?
> Is anyone willing to work on this with me? I am not intimately
> familiar with the scipy codebase, but I have coded up many logpdf
> gradients for PyMC.

I would be very interested in this and similar enhancements (Hessian
of logpdf, hazard function, characteristic function).

Adding a distribution specific _logpdf_gradient (?name) would just
need the derivative with respect to the shape parameters, and can be
directly added to the individual distribution classes.
(I think the only restriction is that it should be vectorized to work
with a 1d array of arguments. ?)

This wouldn't require any knowledge of the generic machinery. So,
forking scipy and starting to edit the individual distributions in
scipy.stats.distributions.py will get you started.


loc and scale are usually handled in the superclass methods, without
the underline in the name.
For this, I think it would be better to come up with a program
friendly interface, instead of a user friendly interface by just
copying the pattern of existing methods.
Since the gradient returns a vector (per observation) the broadcasting
style of the current methods won't work directly, and I'm not a fan of
*args and **kwds, optimization arguments should be an array in my
opinion. Another thought is overhead of the function call since it
will be mainly useful for optimization
http://projects.scipy.org/scipy/ticket/1389 although vectorized and
with a good optimizer convergence should be pretty fast.

The distribution specific methods would be useful independently of the
wrapper, and would also allow other packages to write their own
wrappers. Maybe we can add a dumb wrapper (as private method?) that
just handles loc and scale in the simplest way.

Does PyMC use a generic way of getting the derivative with respect to
loc and scale or is it distribution specific?

----------
In statsmodels, we currently use only distribution and model specific
gradients and hessians of the log-likelihood or numerical gradients.
I started to experiment with a generic system when I worked on the
t-distribution. sympy worked quite well and has the special function
that is required for the gradient of the t-distribution and that I
didn't know about. My plan was to use the derivative with respect to
shapes, loc and scale, and chain them together with other functions
that represent for example loc or scale as function of some parameters
and explanatory variables. It turned out to be a lot more work than I
had time for.
---------

So please go ahead, we should be able to add at least any private
methods pretty fast.
The tests will require just a few lines if we compare them against a
numerical derivative, there is a check_derivative somewhere in
scipy.optimize.

Josef









>
> John
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>