Constructing a mean ROC curve based on 5 iterations
1
0
Entering edit mode
4.3 years ago

I did a 5-fold cross-validation for testing a binary classifier. Using the results, I constructed a ROC curve for each iteration with ROCR package:

enter image description here

Is there any way to construct a mean ROC curve based on these 5 curves?

roc R rocr • 1.7k views
ADD COMMENT
0
Entering edit mode

This seems off-topic for biostars? Perhaps more appropriate to ask on stats.stackexchange.com

ADD REPLY
2
Entering edit mode
4.3 years ago
Collin ▴ 1000

What happens in 5-fold cross-validation is that you train on 4 of the folds and predict on the hold-out fold. This procedure is repeated until all folds have hold-out predictions on them. Given this preamble, you could just concatenate the scores and labels from the 5 hold-out predictions (which should comprise all of your data points), and then construct a single ROC curve based on that.

Ideally with enough data and randomization of which samples fall into each fold, there should not be large discrepancies between the ROC curve of each fold individually.

ADD COMMENT
0
Entering edit mode

What you mean is to join the results of each interation in a single vector, right? The problem is that I am using a program that doesn't give a score of each datapoint, it only sorts the data points. Therefore, I can't join the results maintaining the rank...

ADD REPLY
1
Entering edit mode

Yes, I meant to concatenate them into a single vector. Nearly all methods provide a score which is used for ranking. It's odd that it's not provided as output. They may have an internal score that is just not getting returned to the user. You might need to contact the authors or see if you can modify their code to return the score as an output.

ADD REPLY
0
Entering edit mode

Yes, it's internally calculated but I can't change the code. I'll try to talk with the authors.

ADD REPLY

Login before adding your answer.

Traffic: 1063 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6