[scikit-learn] Plot Cross-validated ROCs for multi-class classification problem

[serafim loukas]

Hello J.B,


I could simply create some ROC curves as shown in the scikit-learn documentation by selecting only 2 classes and then repeating by selecting other pair of classes (in total I have 3 classes so this would result in 3 different ROC figures).

An alternative would be I would like to plot the mean and confidence intervals of the 3-class Cohen Kappa metric as estimated by KFolds (k=5) cross-validation.

Any tips about this ?


Cheers,
Makis



On 21 Jul 2018, at 16:02, Brown J.B. via scikit-learn <scikit-learn at python.org<mailto:scikit-learn at python.org>> wrote:

Hello Makis,

2018-07-20 23:44 GMT+09:00 Andreas Mueller <t3kcit at gmail.com<mailto:t3kcit at gmail.com>>:
There is no single roc curve for a 3 class problem. So what do you want to plot?

On 07/20/2018 10:40 AM, serafim loukas wrote:
What I want to do is to plot the average(mean) ROC across Folds for a 3-class case.

The prototypical ROC curve uses True Positive Rate and False Positive Rate for its axes, so it is for 2-class problems, and not for 3+-class problems, as Andy mentioned.
Perhaps you are wanting the mean and confidence intervals of the n-class Cohen Kappa metric as estimated by either many folds of cross validation, or you want to evaluate your classifier by repeated subsampling experiments and Kappa value distribution/histogram?

Hope this helps,
J.B.
_______________________________________________
scikit-learn mailing list
scikit-learn at python.org<mailto:scikit-learn at python.org>
https://mail.python.org/mailman/listinfo/scikit-learn

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20180721/5a5d4bc2/attachment.html>