Systematic review search strategies are poorly described and not reproducible: a cross-sectional meta-research study

Abstract

ABSTRACTObjectiveTo determine the reproducibility of biomedical systematic review search strategies.DesignCross-sectional meta-research study.PopulationRandom sample of 100 systematic reviews indexed in MEDLINE in November 2021.Main Outcome MeasuresThe primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced. This was operationalized as fulfilling six key PRISMA-S reporting guideline items (database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records) and having all database searches reproduced within 10% of the number of original results.ResultsThe 100 systematic review articles contained 453 database searches. Of those, 214 (47.2%) provided complete database information (named the database and platform; PRISMA-S item 1). Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; 6 searches differed by more than 1000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary details for all database searches to be fully reproducible.ConclusionSystematic review search reporting is poor. As systematic reviews and clinical practice guidelines based upon them continue to proliferate, so does research waste. To correct this will require a multi-faceted response from systematic review authors, peer reviewers, journal editors, and database providers.