Question: Constructing a mean ROC curve based on 5 iterations
0
gravatar for Swimming bird
9 months ago by
Swimming bird10 wrote:

I did a 5-fold cross-validation for testing a binary classifier. Using the results, I constructed a ROC curve for each iteration with ROCR package:

enter image description here

Is there any way to construct a mean ROC curve based on these 5 curves?

rocr roc R • 238 views
ADD COMMENTlink modified 8 months ago by Collin650 • written 9 months ago by Swimming bird10

This seems off-topic for biostars? Perhaps more appropriate to ask on stats.stackexchange.com

ADD REPLYlink written 8 months ago by Len Trigg1.2k
2
gravatar for Collin
8 months ago by
Collin650
United States
Collin650 wrote:

What happens in 5-fold cross-validation is that you train on 4 of the folds and predict on the hold-out fold. This procedure is repeated until all folds have hold-out predictions on them. Given this preamble, you could just concatenate the scores and labels from the 5 hold-out predictions (which should comprise all of your data points), and then construct a single ROC curve based on that.

Ideally with enough data and randomization of which samples fall into each fold, there should not be large discrepancies between the ROC curve of each fold individually.

ADD COMMENTlink modified 8 months ago • written 8 months ago by Collin650

What you mean is to join the results of each interation in a single vector, right? The problem is that I am using a program that doesn't give a score of each datapoint, it only sorts the data points. Therefore, I can't join the results maintaining the rank...

ADD REPLYlink written 8 months ago by Swimming bird10
1

Yes, I meant to concatenate them into a single vector. Nearly all methods provide a score which is used for ranking. It's odd that it's not provided as output. They may have an internal score that is just not getting returned to the user. You might need to contact the authors or see if you can modify their code to return the score as an output.

ADD REPLYlink written 8 months ago by Collin650

Yes, it's internally calculated but I can't change the code. I'll try to talk with the authors.

ADD REPLYlink written 8 months ago by Swimming bird10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 2225 users visited in the last hour