Is Artificial Intelligence Replacing Humans in Systematic Literature Reviews? A Systematic Literature Review

Abstract

Abstract Background Systematic literature reviews (SLRs) seek to answer research questions and form comprehensive, rigorous evidence-based conclusions. However, SLRs are resource-intensive. Multiple SLR applications provide artificial intelligence (AI)-as-a-service capabilities (e.g. EPPI-Reviewer, DistillerSR, Abstrackr) aimed at reducing this workload. The primary objective of our SLR was to assess whether and how researchers employ the AI component of these applications in published SLRs. Methods MEDLINE and EMBASE were searched in June 2021 for SLRs that utilized AI. SLRs and scoping reviews were eligible if they addressed human healthcare-related questions and reported the use of AI or AI-as-a-service applications (AIsAPP) in any step. Rayyan was used for abstract and full-text screening. Backwards citation-tracking and hand-searching were completed. Data were abstracted into Google Sheets and used to summarize key characteristics of included studies in frequency tables and figures. Results In the fifty-six studies included, the most frequently utilized AIsAPPs were Rayyan (n = 22) DistillerSR (n = 11) and EPPI reviewer (n = 9). Only 16 studies provided details on the automation process; thirteen used AI for semi-automation, two for full-automation, and for one study it was not clear how automation was used. Semi-automation was mainly used for screening (n = 12) and extraction (n = 2). Full automation was used for abstract screening in two studies. Conclusions Few SLRs reported utilizing AI in the SLR process. Contrary to recommendations in current PRISMA guidelines, details on automation steps were rarely described. Transparently reporting how AI is used during the conduct of an SLR could aid decision-makers with the adoption of such technologies as standard practice. Further research should evaluate the limitations and barriers to incorporating and reporting the use of AI as standard in SLRs.