[SciPy-User] Unit testing of Bayesian estimator

[Anne Archibald]

2009/11/9 Bruce Southey <bsouthey at gmail.com>:
> On 11/09/2009 12:06 PM, Anne Archibald wrote:
>> 2009/11/9 Bruce Southey<bsouthey at gmail.com>:
>>
>>> I do not know what you are trying to do with the code as it is not my
>>> area. But you are using some empirical Bayesian estimator
>>> (http://en.wikipedia.org/wiki/Empirical_Bayes_method) and thus you lose
>>> much of the value of Bayesian as you are only dealing with modal
>>> estimates. Really you should be obtaining the distribution of
>>> "Probability the signal is pulsed" not just the modal estimate.
>>>
>> Um. Given a data set and a prior, I just do Bayesian hypothesis
>> comparison. This gives me a single probability that the signal is
>> pulsed. You seem to be imagining a probability distribution for this
>> probability - but what would the independent variables be? The
>> unpulsed distribution does not depend on any parameters, and I have
>> integrated over all possible values for the pulsed distribution. So
>> what I get should really be the probability, given the data, that the
>> signal is pulsed. I'm not using an empirical Bayesian estimator; I'm
>> doing the numerical integrations directly (and inefficiently).
>>
> Here are two links on what I mean with reference to the binomial case:
> http://lingpipe-blog.com/2009/09/11/batting-averages-bayesian-vs-mle-estimate/
>
> TEACHING OF BAYESIAN ESTIMATION OF “P” PROBABILITY
> IN A BERNOULLI PROCESS:
> http://www.stat.auckland.ac.nz/~iase/publications/17/C439.pdf
>
>
> I do not know your area but you should be able to do something similar.

They are doing something essentially different from what I am doing.
They have a single (parameterized) hypothesis, so they don't compute a
probability of it being the case rather than some other hypothesis.
Perhaps you are being misled by the fact that the system they are
reasoning about is a binomial system, in which the parameter is
"probability of occurrence". In my case, I am not working with a
binomial system; the closest analog in my system to their p is my
fraction parameter, and I seem to have a usable way to test the
posterior distribution of this parameter. It is the hypothesis testing
that I am trying to test at the moment.

Anne