Is evaluation based on accuracy of classification algorithms misleading? An approach to model validation using Bayes error rate

Document Type : Research Article

Authors

Faculty of Mathematics and Computer Science, Amirkabir University of Technology

10.22124/jmm.2025.28744.2555

Abstract

 Researchers have long regarded model accuracy as the primary metric for evaluating
 the performance of classification algorithms. The current evaluation approach, which relies solely
 on model accuracy, often leads to inappropriate evaluation of classifiers, regardless of the dataset’s
 separability and complexity. This limitation underscores the need for a new and more comprehen
sive method. We argue that accuracy-based evaluation can be misleading, even when considering
 measures of data separability and complexity. We compare the error rates of well-known classifiers
 on Gaussian-generated datasets and show that, paradoxically, many algorithms’ observed errors are
 lower than that of the theoretical optimal classifier, leading to an overestimation of their performance.
 We consider a model invalid if its error rate is lower than the optimal classifier error, known as the
 Bayes error rate. To identify such invalid models, we introduce a procedure and propose an algorithm
 for model validation based on the Bayes error rate.

Keywords

Main Subjects

Journal of Mathematical Modeling
Volume 13, Issue 4
December 2025
Pages 917-927

Files

Share

How to cite

Statistics