Binary search of the optimal cut-point value in ROC analysis using the F1 score

Abstract

Abstract This paper introduces a binary search algorithm for determining the optimal probability cut-point value (C) of binary classifiers. Cut-points are operating points on the receiver operating characteristic curve that divide positive and negative predictions. Compared to the traditional exhaustive search for optimal C value, the proposed method offers execution time efficiency (O(log2(k))) and a small cut-point error of 1/2 n after k steps of binary search. Traditionally, the optimal C value is determined by stepping through all possible C values. This search is uninformed because there is no indication of the search direction. To address this issue, we derive the expectation of the F-Measure (aka F1 score); and use it to guide the search process. Specifically, by comparing the F-Measure at the current cut-point with the F-Measure at expected cut-point, we can use the information to adjust C dynamically towards the optimal cut-point, resulting in optimal model performance. Our results on two classifiers trained from disease classification datasets suggest that the algorithm is robust and efficient, as compared to the traditional methods.