Entering edit mode
9 days ago
ssko
▴
20
Let's say I run the classification model on 100 different data splits, for each data split I collect the predicted probability of the test set. How should I draw a roc curve?
- Should I pool all the probabilities to draw it or
- Should I calculate the ROC/AUC for each data section and then average the ROC/AUC with the std?
I mean, first we split the data set into training and testing, train a specific algorithm for that split on the training set and evaluate the trained algorithm on the test set. I mean to run this process again on 100 different data splits. I want to get a more general result without depending on the performance of the model on a single data split.
I got what you meant, but not convinced that you have the execution part covered.
The way this is normally done is called cross-validation), or CV for short. For a reasonably large dataset, say tens to hundreds of thousands of data points, there is no need to do CV more than 5-10 folds. Only if you have a dataset where the number of data points is in hundreds is it justified to do a 100-fold CV, or even a leave-one-out CV (also described at the link above).