Shortcomings in the Evaluation of Blood Glucose Forecasting (Preprint)

Abstract

Most artificial pancreas systems require a blood glucose (BG) forecasting model that captures the dynamics of the human metabolic system. Machine learning researchers train these models by optimizing for metrics such as root mean squared error (RMSE). However, we found that when combined with a standard controller, models that minimize RMSE do not necessarily yield a higher percent time-in-range (%TIR). We compared the predictive accuracy and control performance of two forecasters: a machine learning-based model that minimizes RMSE (LSTM) and a rule-based model (Loop). Despite achieving RMSE comparable to state-of-the-art (RMSE 15.24mg/dL at 30min), LSTM only achieved 24.35% (IQR 22.35-25.61) TIR. While Loop’s prediction accuracy was worse (RMSE 19.50mg/dL at 30min, p < 0.05), it achieved higher TIR: 34.20% (IQR 31.25-41.02). Thus, the standard approach to evaluating BG forecasters could lead to poor model selection with respect to improving closed-loop control.