ROC Curves, Loss Functions, and Distorted Probabilities in Binary Classification

Abstract

The main purpose of this work is to study how loss functions in machine learning influence the “binary machines”, i.e., probabilistic AI models for predicting binary classification problems. In particular, we show the following results: (i) Different measures of accuracy such as area under the curve (AUC) of the ROC curve, the maximal balanced accuracy, and the maximally weighted accuracy are topologically equivalent, with natural inequalities relating them; (ii) the so-called real probability machines with respect to given information spaces are the optimal machines, i.e., they have the highest precision among all possible machines, and moreover, their ROC curves are automatically convex; (iii) the cross-entropy and the square loss are the most natural loss functions in the sense that the real probability machine is their minimizer; (iv) an arbitrary strictly convex loss function will also have as its minimizer an optimal machine, which is related to the real probability machine by just a reparametrization of sigmoid values; however, if the loss function is not convex, then its minimizer is not an optimal machine, and strange phenomena may happen.