Bayesian inference for uncertainty quantification in binary classification tasks with limited data

Abstract

In recent years, Uncertainty Quantification has gained significant attention as an essential aspect of Machine Learning models, particularly in the context of decision-making and risk assessment. In this study, we provide a comprehensive investigation of the performance of Bayesian Inference for Uncertainty Quantification in binary classification problems with limited data, and we compare its effectiveness with alternative approaches, such as the uncertainty region and Conformal Inference methods. We apply Bayesian logistic regression models, as well as the other two methods, to various well-known small datasets. The performance of these approaches is evaluated using F1 scores at different levels of the uncertainty predicted by each method. Our results demonstrate that the Bayesian Inference approach not only exhibits competitive performance, particularly when using non-informative priors, but also provides a more flexible output in the form of a distribution, this adaptability enables a more comprehensive analysis of the results and can be leveraged in various ways to capture different levels of uncertainty. Despite the convergence challenges associated with small data sets and high-dimensional parameter spaces, the Bayesian models capture both epistemic and aleatoric uncertainties, offering a robust framework for understanding the underlying phenomena. In contrast, the uncertainty region and Conformal Inference methods display certain limitations, such as reduced training set size for Conformal Inference and inflexibility in the out put, which could potentially hinder their practical applications. Our findings highlight the potential of Bayesian Inference as an effective approach for Uncertainty Quantification in binary classification problems, even when faced with limited data.