What is the value of hospital mortality indicators, and are there ways to do better?

Abstract

Monitoring hospital performance using patient safety indicators is one of the key components of healthcare reform in Australia. Mortality indicators, including the hospital standardised mortality ratio and deaths in low mortality diagnosis reference groups have been included in the core national hospital-based outcome indicator set recommended for local generation and review and public reporting. Although the face validity of mortality indicators such as these is high, an increasing number of studies have demonstrated that there are concerns regarding their internal, construct and criterion validity. Use of indicators with poor validity has the consequence of potentially incorrectly classifying hospitals as performance outliers and expenditure of limited hospital staff time on activities which may provide no gain to hospital quality and safety and may in fact cause damage to morale. This paper reviews the limitations of current approaches to monitoring hospital quality and safety performance using mortality indicators. It is argued that there are better approaches to improving performance than monitoring with mortality indicators generated from hospital administrative data. These approaches include use of epidemiologically sound, clinically relevant data from clinical-quality registries, better systems of audit, evidence-based bundles, checklists, simulators and application of the science of complex systems. What is known about the topic? Public reporting of adverse events such as hospital standardised mortality ratios deaths in low mortality diagnosis reference groups is a key component of Australian healthcare reform. There is much debate in Australia and internationally concerning the appropriateness of this approach. What does the paper add? We extend the current literature and debate by reviewing the statistical limitations, challenges and biases inherent in these indicators. Alternatives for quality and safety performance monitoring that are more robust are presented. What are the implications for practitioners? The hospital standardised mortality ratio and death in low mortality diagnosis reference groups indicators should be used with extreme caution. Although public reporting of quality and safety indicators is necessary there are likely to be better methods to detect substandard performance. These include: properly structured morbidity and mortality meetings, independent audits, evidence-based bundles and checklists, sequential data analysis (e.g. using CUSUMS), and the use of simulators. To achieve maximum safety it is necessary, in addition to using these methods, to understand the characteristics of hospitals as complex systems that exhibit safe emergent behaviour, e.g. using the science of complex systems and its tools. Genuine safety cannot be achieved simply be studying ‘unsafety’. In addition, epidemiologically sound, clinically relevant clinical-quality registries are required.