[scikit-learn] AUCROC/MAP confidence intervals in scikit

[Sebastian Raschka]

Still haven't had a chance to read it, but ROC for binary classification anyway? Also, i.i.d. assumptions are typical for the learning algorithms as well.

Best,
Sebastian

> On Feb 7, 2019, at 10:15 AM, josef.pktd at gmail.com wrote:
> 
> Just a skeptical comment from a bystander.
> 
> I only skimmed parts of the article. My impression is that this does not apply (directly) to the regression setting.
> AFAIU, they assume that all observations have the same propability.
> 
> To me it looks more like the literature on testing of or confidence intervals for a single proportion.
> 
> I might be wrong.
> 
> Josef
> 
> On Thu, Feb 7, 2019 at 11:00 AM Andreas Mueller <t3kcit at gmail.com> wrote:
> The paper definitely looks interesting and the authors are certainly 
> some giants in the field.
> But it is actually not widely cited (139 citations since 2005), and I've 
> never seen it used.
> 
> I don't know why that is, and looking at the citations there doesn't 
> seem to be a lot of follow-up work.
> I think this would need more validation before getting into sklearn.
> 
> Sebastian: This paper is distribution independent and doesn't need 
> bootstrapping, so it looks indeed quite nice.
> 
> 
> On 2/6/19 1:19 PM, Sebastian Raschka wrote:
> > Hi Stuart,
> >
> > I don't think so because there is no standard way to compute CI's. That goes for all performance measures (accuracy, precision, recall, etc.). Some people use simple binomial approximation intervals, some people prefer bootstrapping etc. And it also depends on the data you have. In large datasets, binomial approximation intervals may be sufficient and bootstrapping too expensive etc.
> >
> > Thanks for sharing that paper btw, will have a look.
> >
> > Best,
> > Sebastian
> >
> >
> >> On Feb 6, 2019, at 11:28 AM, Stuart Reynolds <stuart at stuartreynolds.net> wrote:
> >>
> >> https://papers.nips.cc/paper/2645-confidence-intervals-for-the-area-under-the-roc-curve.pdf
> >> Does scikit (or other Python libraries) provide functions to measure the confidence interval of AUROC scores? Same question also for mean average precision.
> >>
> >> It seems like this should be a standard results reporting practice if a method is available.
> >>
> >> - Stuart
> >> _______________________________________________
> >> scikit-learn mailing list
> >> scikit-learn at python.org
> >> https://mail.python.org/mailman/listinfo/scikit-learn
> > _______________________________________________
> > scikit-learn mailing list
> > scikit-learn at python.org
> > https://mail.python.org/mailman/listinfo/scikit-learn
> 
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn