Integrating economic considerations into cutpoint selection may help align clinical decision support toward value-based healthcare

Abstract

AbstractObjectiveClinical prediction models providing binary categorizations for clinical decision support require the selection of a probability threshold, or “cutpoint,” to classify individuals. Existing cutpoint selection approaches typically optimize test-specific metrics, including sensitivity and specificity, but overlook the consequences of correct or incorrect classification. We introduce a new cutpoint selection approach considering downstream consequences using net monetary benefit (NMB) and through simulations compared it with alternative approaches in 2 use-cases: (i) preventing intensive care unit readmission and (ii) preventing inpatient falls.Materials and methodsParameter estimates for costs and effectiveness from prior studies were included in Monte Carlo simulations. For each use-case, we simulated the expected NMB resulting from the model-guided decision using a range of cutpoint selection approaches, including our new value-optimizing approach. Sensitivity analyses applied alternative event rates, model discrimination, and calibration performance.ResultsThe proposed approach that considered expected downstream consequences was frequently NMB-maximizing compared with other methods. Sensitivity analysis demonstrated that it was or closely tracked the optimal strategy under a range of scenarios. Under scenarios of relatively low event rates and discrimination that may be considered realistic for intensive care (prevalence = 0.025, area under the receiver operating characteristic curve [AUC] = 0.70) and falls (prevalence = 0.036, AUC = 0.70), our proposed cutpoint method was either the best or similar to the best of the compared methods regarding NMB, and was robust to model miscalibration.DiscussionOur results highlight the potential value of conditioning cutpoints on the implementation setting, particularly for rare and costly events, which are often the target of prediction model development research.ConclusionsThis study proposes a cutpoint selection method that may optimize clinical decision support systems toward value-based care.