Evaluation methods, indicators, and outcomes in learning health systems: a jurisdictional scan (Preprint)

Abstract

In learning health systems (LHS), real-time evidence, informatics, patient-provider partnerships and experiences, and organizational culture are combined to conduct “learning cycles” that support improvements in care. Though the concept of LHS is fairly well established in the literature, evaluation methods, mechanisms, and indicators are less consistently described. Further, LHS often use “usual care” or “status quo” as a benchmark for comparing new approaches to care, but disentangling usual care from multifarious care modalities found across settings is challenging. There is a need to identify which evaluation methods are used within LHS, describe how LHS growth and maturity are conceptualized, and determine what tools and measures are being used to evaluate LHS at the system level.

To 1) identify international examples of LHS and describe their evaluation approaches, frameworks, indicators and outcomes; and 2) describe common characteristics, emphases, assumptions, or challenges in establishing counterfactuals in LHS.

A jurisdictional scan will be conducted according to modified PRISMA guidelines. LHS will be identified through a search of peer-reviewed and grey literature using Ovid Medline, Ebsco CINAHL, Ovid Embase, Clarivate Web of Science, PubMed Non-Medline databases, and the web. To gain a comprehensive understanding of each LHS, including details specific to evaluation, self-identified LHS will be included if they are described according to ≥4 of 11 pre-specified criteria (core functionalities, analytics, use of evidence, co-design/implementation, evaluation, change management/governance structures, data sharing, knowledge sharing, training/capacity building, equity, sustainability). Search results will be screened, extracted, and analyzed to inform two descriptive reviews pertaining to our two main objectives. Evaluation methods and approaches, both within learning cycles and at the system level, as well as frameworks, indicators, and target outcomes will be identified and summarized descriptively. Across evaluations, common challenges, assumptions, contextual factors, and mechanisms will be described.

Not available.

This research will characterize the current landscape of LHS evaluation approaches and provide a foundation for developing consistent and scalable metrics of LHS growth, maturity, and success.