Stacking multiple prediction models to optimise performance in local settings: exemplars in cardiometabolic disease

Abstract

AbstractBackgroundRisk prediction models are used in healthcare settings to tailor therapies to individuals most likely to benefit. Despite appropriate external validation, difference in local characteristics (e.g. patient mix) may attenuate model performance. Prior to any implementation it is therefore advisable to explore local performance, typically requiring a modest amount of historic data. Depending on model performance, model adjustments might be necessary which often require large amounts of data. Here we explore a small sample size approach approximating de novo derivation, by combining model stacking and transfer learning, referred to asstacked transfer learning. As an example we focus on stacking previously trained risk prediction models for cardiovascular disease (CVD), stroke, (chronic) kidney disease, and diabetes.MethodsWe leverage data from the UK biobank to illustrate the benefits of stacking previously trained risk prediction models, predicting the risk of incident CVD, chronic kidney disease (CKD) or diabetes. To mimic sample sizes available in local settings, such as a small to large healthcare trust, we iterated the number of training cases between 10 and 1000. Model stacking was performed using a LASSO penalized logistic regression model, and compared performance of ade novomodel estimating the local association of 33 variables used in the aforementioned risk prediction models.ResultsWe found that stacked models require roughly one-tenths of the training sample size compared to de novo derivation of a prediction model. For example, predicting CVD the stacked model required 30 cases to reach a area under the curve (AUC) value (with 95% CI) of 0.732 (0.728, 0.735), while thede novomodel required 300 cases to reach approximately the same performance. As expected, the absolute performance depended on the predicted outcome, where for example the difference betweende novoand stacked modelling was smaller for CKD prediction.ConclusionWe show that our proposed ”stacked transfer learning” approach closely approximated the predictive performance of ade novomodel, often requiring only a fraction of the data. As such, this approach should be considered when tailoring a model to a local setting.