The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: a retrospective pilot study

Abstract

Abstract Background Evidence synthesis is valuable but resource and time consuming. This is problematic because syntheses must be updated with new evidence that is published at an increasing rate. Machine learning (ML) tools may accelerate evidence synthesis production, but little is known about their effectiveness with respect to resource use and time to review completion. Methods We obtained data retrospectively from our department at the Norwegian Institute of Public Health (NIPH). We included all analyzable systematic reviews and health technology assessments commissioned between 1 August 2020 (commission of the first review to use ML at NIPH) and 31 January 2023 (study cut-off). The outcomes were time-to-completion (from commission to approval for delivery) and resource use (person hours). The co-primary analyses compared no ML use to recommended ML use. Recommended use was defined as ML use in any review phase consistent with the ML team’s guidance or direct recommendation. We also compared non-recommended ML to recommended ML use, and no ML use to any ML use. We estimated relative time-to-completion and relative resource use, accounting for endogenous treatment assignment and censoring (ongoing reviews). All work was prespecified and, except as described, performed according to a published peer-reviewed protocol. Results We anticipated including about 100 reviews but could only obtain analyzable data from 39. For the co-primary analyses, we estimate that reviews that use ML as recommended require 3.71 (95% CI 0.36 to 37.95; p = 0.269) times as much resource and can be completed in 92% (95% CI 53–158%; p = 0.753) of the time required by reviews that do not use ML as recommended. Conclusion Due to the smaller than anticipated sample size, this pilot study was not able to estimate any of the effects with sufficient precision to conclude that recommended or any ML use is associated with more or less resource use, or longer or shorter time-to-completion, compared to no or non-recommended ML use. We suggest future studied be powered to detect reductions of at least 30% in resource use and 10% in time-to-completion.