Question: Constructing a mean ROC curve based on 5 iterations
0
gravatar for Swimming bird
17 months ago by
Swimming bird20 wrote:

I did a 5-fold cross-validation for testing a binary classifier. Using the results, I constructed a ROC curve for each iteration with ROCR package:

enter image description here

Is there any way to construct a mean ROC curve based on these 5 curves?

rocr roc R • 404 views
ADD COMMENTlink modified 17 months ago by Collin680 • written 17 months ago by Swimming bird20

This seems off-topic for biostars? Perhaps more appropriate to ask on stats.stackexchange.com

ADD REPLYlink written 17 months ago by Len Trigg1.3k
2
gravatar for Collin
17 months ago by
Collin680
United States
Collin680 wrote:

What happens in 5-fold cross-validation is that you train on 4 of the folds and predict on the hold-out fold. This procedure is repeated until all folds have hold-out predictions on them. Given this preamble, you could just concatenate the scores and labels from the 5 hold-out predictions (which should comprise all of your data points), and then construct a single ROC curve based on that.

Ideally with enough data and randomization of which samples fall into each fold, there should not be large discrepancies between the ROC curve of each fold individually.

ADD COMMENTlink modified 17 months ago • written 17 months ago by Collin680

What you mean is to join the results of each interation in a single vector, right? The problem is that I am using a program that doesn't give a score of each datapoint, it only sorts the data points. Therefore, I can't join the results maintaining the rank...

ADD REPLYlink written 17 months ago by Swimming bird20
1

Yes, I meant to concatenate them into a single vector. Nearly all methods provide a score which is used for ranking. It's odd that it's not provided as output. They may have an internal score that is just not getting returned to the user. You might need to contact the authors or see if you can modify their code to return the score as an output.

ADD REPLYlink written 17 months ago by Collin680

Yes, it's internally calculated but I can't change the code. I'll try to talk with the authors.

ADD REPLYlink written 17 months ago by Swimming bird20
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1335 users visited in the last hour