Reproducibility and Robustness of Localized Mortality Prediction

Abstract

AbstractBackgroundWhile localized modeling—the use of predictive models to perform the adaptation step in case-based reasoning—has been evaluated in several experimental settings, its reported successes have infrequently been independently and externally validated.ObjectiveWe aimed to extend and validate an experimental study of mortality prediction in a critical care population and to assess the importance of several methodological factors to predictive performance.MethodsWe reproduced the workflow of Lee, Maslove, and Dubin (2015) using an updated database. We evaluated performance as area under the receiver operating characteristic curve and under the precision–recall curve and calibration as weakness of evidence in Hosmer– Lemeshow tests. We compared the effects of several modeling choices, including how relevance is quantified, and how relevance cohorts are retrieved, and the choice of model. We compared ours to previous results and used linear regression to quantify the role of each modeling choice on performance.ResultsOverall performance and its relationship to cohort size validated previous results. These relationships varied by model family as expected, though we observed no advantage of decision trees over a model-free approach and poor performance by random forests. An alternate choice of unlearned similarity measure yielded marginal and inconsistent performance differences. Denominating cohorts by similarity threshold rather than by cardinality yielded marginal but consistent performance losses. A temporal validation exercise enabled by a change of information system before the recent upgrade corroborated performance estimates from cross-validation. In all, cohort denomination mattered more to performance than any other methodological choice.DiscussionThe greater impact of retrieval than of adaptation suggests a weakness with the strategy of localized modeling. Additional research to deconstruct the varieties of this approach and quantify the relative benefits of its components is needed to resolve this question.