Can we trust the standardized mortality ratio? A formal analysis and evaluation based on axiomatic requirements

Abstract

Background The standardized mortality ratio (SMR) is often used to assess and compare hospital performance. While it has been recognized that hospitals may differ in their SMRs due to differences in patient composition, there is a lack of rigorous analysis of this and other—largely unrecognized—properties of the SMR. Methods This paper proposes five axiomatic requirements for adequate standardized mortality measures: strict monotonicity (monotone relation to actual mortality rates), case-mix insensitivity (independence of patient composition), scale insensitivity (independence of hospital size), equivalence principle (equal rating of hospitals with equal actual mortality rates in all patient groups), and dominance principle (better rating of unambiguously better performing hospitals). Given these axiomatic requirements, effects of variations in patient composition, hospital size, and actual and expected mortality rates on the SMR were examined using basic algebra and calculus. In this regard, we distinguished between standardization using expected mortality rates derived from a different dataset (external standardization) and standardization based on a dataset including the considered hospitals (internal standardization). The results were illustrated by hypothetical examples. Results Under external standardization, the SMR fulfills the axiomatic requirements of strict monotonicity and scale insensitivity but violates the requirement of case-mix insensitivity, the equivalence principle, and the dominance principle. All axiomatic requirements not fulfilled under external standardization are also not fulfilled under internal standardization. In addition, the SMR under internal standardization is scale sensitive and violates the axiomatic requirement of strict monotonicity. Conclusions The SMR fulfills only two (none) out of the five proposed axiomatic requirements under external (internal) standardization. Generally, the SMRs of hospitals are differently affected by variations in case mix and actual and expected mortality rates unless the hospitals are identical in these characteristics. These properties hamper valid assessment and comparison of hospital performance based on the SMR.