[scikit-learn] AUCROC/MAP confidence intervals in scikit

[josef.pktd at gmail.com]

Just a skeptical comment from a bystander.

I only skimmed parts of the article. My impression is that this does not
apply (directly) to the regression setting.
AFAIU, they assume that all observations have the same propability.

To me it looks more like the literature on testing of or confidence
intervals for a single proportion.

I might be wrong.

Josef

On Thu, Feb 7, 2019 at 11:00 AM Andreas Mueller <t3kcit at gmail.com> wrote:

> The paper definitely looks interesting and the authors are certainly
> some giants in the field.
> But it is actually not widely cited (139 citations since 2005), and I've
> never seen it used.
>
> I don't know why that is, and looking at the citations there doesn't
> seem to be a lot of follow-up work.
> I think this would need more validation before getting into sklearn.
>
> Sebastian: This paper is distribution independent and doesn't need
> bootstrapping, so it looks indeed quite nice.
>
>
> On 2/6/19 1:19 PM, Sebastian Raschka wrote:
> > Hi Stuart,
> >
> > I don't think so because there is no standard way to compute CI's. That
> goes for all performance measures (accuracy, precision, recall, etc.). Some
> people use simple binomial approximation intervals, some people prefer
> bootstrapping etc. And it also depends on the data you have. In large
> datasets, binomial approximation intervals may be sufficient and
> bootstrapping too expensive etc.
> >
> > Thanks for sharing that paper btw, will have a look.
> >
> > Best,
> > Sebastian
> >
> >
> >> On Feb 6, 2019, at 11:28 AM, Stuart Reynolds <stuart at stuartreynolds.net>
> wrote:
> >>
> >>
> https://papers.nips.cc/paper/2645-confidence-intervals-for-the-area-under-the-roc-curve.pdf
> >> Does scikit (or other Python libraries) provide functions to measure
> the confidence interval of AUROC scores? Same question also for mean
> average precision.
> >>
> >> It seems like this should be a standard results reporting practice if a
> method is available.
> >>
> >> - Stuart
> >> _______________________________________________
> >> scikit-learn mailing list
> >> scikit-learn at python.org
> >> https://mail.python.org/mailman/listinfo/scikit-learn
> > _______________________________________________
> > scikit-learn mailing list
> > scikit-learn at python.org
> > https://mail.python.org/mailman/listinfo/scikit-learn
>
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20190207/3b6812e9/attachment.html>