Assessing the accuracy of forecasting: Applying standard diagnostic assessment tools to a health technology early warning system

Abstract

Objectives:Early warning systems are an integral part of many health technology assessment programs. Despite this finding, to date, there have been no quantitative evaluations of the accuracy of predictions made by these systems. We report a study evaluating the accuracy of predictions made by the main United Kingdom early warning system.Methods:As prediction of impact is analogous to diagnosis, a method normally applied to determine the accuracy of diagnostic tests was used. The sensitivity, specificity, and predictive values of the National Horizon Scanning Centre's prediction methods were estimated with reference to an (imperfect) gold standard, that is, expert opinion of impact 3 to 5 years after prediction.Results:The sensitivity of predictions was 71 percent (95 percent confidence interval [CI], 0.36–0.92), and the specificity was 73 percent (95 percent CI, 0.64–0.8). The negative predictive value was 98 percent (95 percent CI, 0.92–0.99), and the positive predictive value was 14 percent (95 percent CI, 0.06–0.3).Conclusions:Forecasting is difficult, but the results suggest that this early warning system's predictions have an acceptable level of accuracy. However, there are caveats. The first is that early warning systems may themselves reduce the impact of a technology, as helping to control adoption and diffusion is their main purpose. The second is that the use of an imperfect gold standard may bias the results. As early warning systems are viewed as an increasingly important component of health technology assessment and decision making, their outcomes must be evaluated. The method used here should be investigated further and the accuracy of other early warning systems explored.