Matrix of Evidence Tool: Assessing the quality of real-world study designs (Preprint)

Abstract

There are a variety of study designs that utilize real-world data. To date, there are limited resources available to aid in the selection and evaluation of these study designs to answer important research and policy questions and differentiating between study designs may be challenging for those new to the field of outcomes research.

Therefore, an interactive tool was built based on published methodologic evidence to fill this gap.

The published literature review was evaluated to identify publications related to study designs that utilize real-world data: specifically, pragmatic trials, prospective and retrospective observational studies, and cross-sectional designs. Data from these publications were extracted by two reviewers and qualitatively synthesized to identify themes. These themes were then assembled to build an interactive tool for researchers and decision makers of studies using real-world data. This tool underwent external evaluation and revision in an iterative process to build the final Matrix of Evidence Tool.

The major themes (and subthemes) identified from the literature included data issues (access and data quality), generalizability, outcomes (usability for decision making, long-term outcomes, rare events, specificity of measurement, causal inference), operational issues, bias (compliance, misclassification of exposure, misclassification of outcome, participant selection bias, immortal time bias), and confounding. The Matrix of Evidence tool was built to communicate these themes and is a resource that can be used to evaluate the potential of a particular real-world study design to maximize various aspects of research.

The key themes identified in the published literature can be evaluated across real-world designs by using the Matrix of Evidence tool. While all aspects of real-world research are not feasible to include, the primary differentiating factors across four common real-world designs are evaluated and compared. Researchers may find this tool valuable in the selection of an appropriate real-world study design given the research question and intent of use of the resulting evidence.