Author:
Wessler Benjamin S.,Nelson Jason,Park Jinny G.,McGinnes Hannah,Gulati Gaurav,Brazil Riley,Van Calster Ben,van Klaveren D.,Venema Esmee,Steyerberg Ewout,Paulus Jessica K.,Kent David M.
Abstract
AbstractBackgroundThere are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested and how well they generally perform has not been broadly evaluated.MethodsA SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts PACE CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data.Results2030 external validations of 1382 CPMs were identified. 807 (58%) of the CPMs in the Registry have never been externally validated. On average there were 1.5 validations per CPM (range 0-94). The median external validation AUC was 0.73 (25th −75th percentile [IQR] 0.66, 0.79), representing a median percent decrease in discrimination of −11.1% (IQR −32.4%, +2.7%) compared to performance on derivation data. 81% (n = 1333) of validations reporting AUC showed discrimination below that reported in the derivation dataset. 53% (n = 983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR −13.2, 3.1) for ‘closely related’ validations (n=123), −9.0 (IQR −27.6, 3.9) for ‘related validations’ (n=862) and −17.2% (IQR −42.3, 0) for ‘distantly related’ validations (n=717) (p<0.001).ConclusionMany published cardiovascular CPMs have never been externally validated and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.
Publisher
Cold Spring Harbor Laboratory
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. External Validation of the SYNTAX Score II 2020;Journal of the American College of Cardiology;2021-09