Assigning weights to Classifiers based on AUC values

Question

[siu]

Hi all,

To identify remote homology, I have used different tools using benchmark dataset for which I have the labels for the true homologs. I have calculated the AUC values based on True positives and False positives in the homology result for each classifier.

The AUC values for different classifiers are like this

Classifier 1 -0.601 
Classifier 2 - 0.732 
Classifier 3 - 0.752
Classifier 4  - 0.828 
Classifier 5 - 0.883

What is the best way to assign weights to each classifier (as they have different performance) so that I can multiply these weights to the normalized scores of each individual tools to find the aggregated score?

Thanks in advance

Answer 1

[Mensur Dlakic]

What is the best way to assign weights to each classifier

It depends on what quantity you are trying to optimize. If you are interested in maximum AUC value in the end, the weights should be based on that quantity. Maximum AUC doesn't necessarily mean maximum accuracy, for example. Nor does it mean minimum log-loss value.

AUC is a ranking-based measure that in general works the best for imbalanced datasets. The threshold for determining true/false homologs may not be at probability=0.5, but it could be at p=0.8 instead.

[siu]

Thanks for your answer. I am interested in giving weights to each tool that I have used for detecting the homologs for my query proteins as stated in "https://www.nature.com/articles/srep32333#Sec6".

I am unable to find that how are they assigning weights to each tool.