Evaluation of the EsteR Toolkit for COVID-19 Decision Support: Sensitivity Analysis and Usability Study (Preprint)

Abstract

In the COVID-19 pandemic, local health authorities are responsible for managing and reporting the current cases in Germany. Since March 2020, the employees had to focus on the pandemic. In the EsteR project, we implemented existing and newly developed statistical models in decision support tools to assist the work in the local health authorities.

The two main goals of this work are investigating the stability of the answers provided by our statistical tools regarding model parameters and evaluating the usability and applicability of our web application.

For the model stability assessment, a sensitivity analysis was carried out for all five statistical models. The default parameters of our models and the ranges the parameters were tested in were based on a prior literature review on COVID-19 properties. For the usability evaluation of the web application, cognitive walkthroughs and focus group interviews were conducted with employees of two different local health authorities.

The simulation results showed that some statistical models are more sensitive to changes in their parameters than others. For each of the single person use cases we were able to find an area where we rate the respective model to be stable, whereas for the group use cases the stability highly depends on the user inputs. The cognitive walkthroughs and the focus group interview disclosed that the user interface had to be simplified and more information was needed as guidance. In general, the testers rated the web application as helpful, especially for new employees.

This evaluation study allowed for a refinement of our toolkit. With the simulations, we identified suitable model parameters and analyzed how stable the statistical models are regarding changes in their parameters. Furthermore, the web application was improved with the results of the cognitive walkthroughs and focus group interviews.