A scoping review of the use of log data for evaluating mobile Apps (Preprint)

Abstract

There is a growing trend in the potential benefits and application of log data for the evaluation of mHealth Apps. However, the process by which insights may be derived from log data remains unstructured, resulting in underutilisation of mHealth data.

We aimed to acquire an understanding of how log data analysis can be used to generate valuable insights in support of realistic evaluations of mobile Apps through a scoping review. This understanding is delineated according to publication trends, associated concepts and characteristics of log data, framework or processes required to develop insights from log data, and how these insights may be utilised towards evaluation of Apps.

The PRISMA-ScR guidelines for a scoping review were followed. The Scopus database, the Journal of Medical Internet Research (JMIR), and grey literature (through a Google search) delivered 105 articles of which 33 articles were retained in the sample for analysis and synthesis.

A definition for log data is developed from its characteristics and articulated as: anonymous records of users’ real-time interactions with the application, collected objectively or automatically and often accessed from cloud-based storage. Publications for theoretical and empirical work on log data analysis have increased between 2010 and 2021 (100% and 95% respectively). The research approach is distributed between inductive (43%), deductive (30%), and a hybrid approach (27%). Research methods include mixed-methods (73%) and quantitative only (27%), although mixed-methods dominate since 2018. Only 30% of studies articulated the use of a framework or model to perform the log data analysis. Four main focus areas for log data analysis are identified as usability (40%), engagement (15%), effectiveness (15%), and adherence (15%). An average of one year of log data is used for analysis, with an average of three years from the launch of the App to the analysis. Collected indicators include user events or clicks made, specific features of the App, and timestamps of clicks. The main calculated indicators are features used or not used (24/33), frequency (21/33), and duration (18/33). Reporting the calculated indicators per ‘user or user group’ was the most used reference point.

Standardised terminology, processes, frameworks, and explicit benchmarks to utilise log data are lacking in literature. Thereby, the need for a conceptual framework that is able to standardise the log analysis of mobile Apps is determined. We provide a summary of concepts towards such a framework.

NA