Comparison of AUCs of two correlated ROC curves for different independent prediction variables

Question

[Biostars2200]

Hello, I'm not sure if this question is appropriate here.

I have done ROC curve analysis.

Different single parameters (continuous variable) were examined for prediction of the same outcome event (Good or Poor) in the same dataset.

I've compared AUCs from those ROC curves, using DeLong test.

However, there was an opinion from my colleague that DeLong test might be inappropriate for this comparison because this is a comparison of nested models.

The analysis was done as below.

using the dataset, aSAH

data(aSAH)

head(aSAH)

ROC curve analysis for different single parameters (i.e. s100, ndka)

roc1<-roc(aSAH$outcome, aSAH$s100)

roc2<-roc(aSAH$outcome, aSAH$ndka)

Comparison of AUCs from two correlated ROC curves for different individual prediction variable

roc.test(roc1, roc2, method="delong")

The plot displaying the two ROC curves with

roc1<-plot.roc(aSAH$outcome, aSAH$s100, percent=TRUE, col="#1c61b6")

roc2<-lines.roc(aSAH$outcome, aSAH$ndka, percent=TRUE, col="#008600")

test<-roc.test(roc1, roc2)

text(50, 50, labels=paste("p-value =", format.pval(test$p.value)), adj=c(0, .5))

In my opinion, this analysis is valid and is not a comparison of nested models, since this is just a comparison of prediction power of single variables.

I hope to listen to any comments about this matter.

Thank you.

Answer 1

[Asaf]

I was about to formulate a lengthy answer but then I found this paper which specifically answers your question. To summarize it: The null hypothesis is different when you're comparing nested and unrelated models and specific tests like LRT are less sensitive to the performances of the nested model.

[Biostars2200]

Thank you for the advice and reference!