Risk of bias and open science practices in systematic reviews of educational effectiveness: A meta‐review

Abstract

AbstractIn order to produce the most reliable syntheses of the effectiveness of educational interventions, systematic reviews need to adhere to rigorous methodological standards. This meta‐review investigated risk of bias occurring while conducting a systematic review and the presence of open science practices like data sharing and reproducibility of the review procedure, in recently published reviews in education. We included all systematic reviews of educational interventions, instructions and methods for all K‐12 student populations in any school form with experimental or quasi‐experimental designs (an active manipulation of the intervention) with comparisons and where the outcome variables were academic performance of any kind. We searched the database Education Resources Information Center (ERIC) through the years 2019–2021. In parallel we hand‐searched four major educational review journals for systematic reviews: Educational Research Review (Elsevier), Educational Review (Taylor & Francis), Review of Education (Wiley), and Review of Educational Research (AERA). Systematic reviews were assessed with the risk of bias tool ROBIS and whether the studies had pre‐registered protocols, shared primary research data, and whether a third party could reproduce search strings and details of where exactly primary research data were extracted. A total of 88 studies that matched our PICOS were included in this review; of these, 10 educational systematic reviews were judged as low risk of bias (approximately 11%) . The rest were classified as high risk of bias during a shortened ROBIS assessment or assessed as high risk or unclear risk of bias following a full ROBIS assessment. Of the 10 low risk of bias reviews, 6 had detailed their search sufficiently enough for a third party to reproduce, 3 reviews shared the data from primary studies, however none had specified how and from where exactly data from primary studies were extracted. The study shows that at least a small part of systematic reviews in education has a low risk of bias, but most systematic reviews in our set of studies have high risk of bias in their methodological procedure. There are still improvements in this field to be expected as even the low risk of bias reviews are not consistent regarding pre‐registered protocols, data sharing, reproducibility of primary research data and reproducible search strings. Context and implications Rationale for this study Rigorous systematic review is the method to use to evaluate and provide synthesis of methods and interventions in educational research. Why the new findings matter Alarmingly few systematic reviews were judged with low risk of bias and few reviews utilise recent open science practices. This might undermine the reliability of study findings. Implications for journals and review researchers This meta‐review highlights the need for review researchers and journals to better adopt best practices in systematic reviews. We therefore urge for better dissemination and awareness of systematic reviews standards and for more transparency in the review process, which may lead to more reliable synthesis of what works best in education. In the long run this is a necessary change and way forward to provide evidence‐based practices in the classroom.