Validation and Calibration of the Risk Stratification Index

Abstract

Abstract Background The Risk Stratification Index was developed from 35 million Medicare hospitalizations from 2001 to 2006 but has yet to be externally validated on an independent large national data set, nor has it been calibrated. Finally, the Medicare Analysis and Provider Review file now allows 25 rather than 9 diagnostic codes and 25 rather than 6 procedure codes and includes present-on-admission flags. The authors sought to validate the index on new data, test the impact of present-on-admission codes, test the impact of the expansion to 25 diagnostic and procedure codes, and calibrate the model. Methods The authors applied the original index coefficients to 39,753,036 records from the 2007–2012 Medicare Analysis data set and calibrated the model. The authors compared their results with 25 diagnostic and 25 procedure codes, with results after restricting the model to the first 9 diagnostic and 6 procedure codes and to codes present on admission. Results The original coefficients applied to the 2007–2012 data set yielded C statistics of 0.83 for 1-yr mortality, 0.84 for 30-day mortality, 0.94 for in-hospital mortality, and 0.86 for median length of stay—values nearly identical to those originally reported. Calibration equations performed well against observed outcomes. The 2007–2012 model discriminated similarly when codes were restricted to nine diagnostic and six procedure codes. Present-on-admission models were about 10% less predictive for in-hospital mortality and hospital length of stay but were comparably predictive for 30-day and 1-yr mortality. Conclusions Risk stratification performance was largely unchanged by additional diagnostic and procedure codes and only slightly worsened by restricting analysis to codes present on admission. The Risk Stratification Index, after calibration, thus provides excellent discrimination and calibration for important health services outcomes and thus appears to be a good basis for making hospital comparisons.