A Machine Learning–Enabled Partially Observable Markov Decision Process Framework for Early Sepsis Prediction

Abstract

Sepsis is a life-threatening condition, caused by the body’s extreme response to an infection. In the United States, 1.7 million cases of sepsis occur annually, resulting in 265,000 deaths. Delayed diagnosis and treatment are associated with higher mortality rates. An exponential rise in the availability of medical data has allowed for the development of sophisticated machine learning algorithms to predict sepsis earlier than the onset. However, these models often underperform, as the training data are retrospective and do not fully capture the uncertain future. In this study, we develop a novel framework, which we refer to as MLePOMDP, to leverage and combine the underlying, high-level knowledge about sepsis progression and machine learning (ML) for classification. Specifically, we use a hidden Markov model to describe sepsis development at a high level, where the ML model makes the higher-order “observations” from temporal data. Consequently, a partially observable Markov decision process (POMDP) model is developed to make classification decisions. We analytically establish that the optimal policy is of threshold-type, which we exploit to efficiently optimize MLePOMDP. MLePOMDP is calibrated and tested using high-frequency physiological data collected from bedside monitors. Different from past POMDP-based frameworks, MLePOMDP is developed for a prediction task using a very small state definition, produces highly interpretable results, and accounts for a novel and clinically meaningful action space. Our results show that MLePOMDP outperforms machine learning–based benchmarks by up to 8% in precision. Importantly, MLePOMDP is able to reduce false alarms by up to 28%. An additional experiment is conducted to show the generalizability of MLePOMDP to different patient cohorts. Summary of Contribution: This study develops a novel real-time decision support framework for early sepsis prediction by integrating well-known machine learning models (random forest and neural networks) with a well-established sequential decision-making model, namely, a partially observable Markov decision process (POMDP). The structural properties of the optimal policy are further explored and a threshold-type structure is established, which is then leveraged to develop a customized algorithm to solve the problem more efficiently. The resulting framework demonstrates the benefit of applying POMDPs to augment machine learning outputs. Specifically, the framework results in the reduction of false alarms in sepsis predictions where decisions are made in real time, hence improving the overall prediction precision.