Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions

Abstract

Systematic reviews (SR) are crucial in synthesizing and analyzing existing scientific literature to inform evidence-based decision-making. However, traditional SR methods often have limitations, including a lack of automation and decision support, resulting in time-consuming and error-prone reviews. To address these limitations and drive the field forward, we harness the power of the revolutionary language model, ChatGPT, which has demonstrated remarkable capabilities in various scientific writing tasks. By utilizing ChatGPT’s natural language processing abilities, our objective is to automate and streamline the steps involved in traditional SR, explicitly focusing on literature search, screening, data extraction, and content analysis. Therefore, our methodology comprises four modules: (1) Preparation of Boolean research terms and article collection, (2) Abstract screening and articles categorization, (3) Full-text filtering and information extraction, and (4) Content analysis to identify trends, challenges, gaps, and proposed solutions. Throughout each step, our focus has been on providing quantitative analyses to strengthen the robustness of the review process. To illustrate the practical application of our method, we have chosen the topic of IoT applications in water and wastewater management and quality monitoring due to its critical importance and the dearth of comprehensive reviews in this field. The findings demonstrate the potential of ChatGPT in bridging the gap between traditional SR methods and AI language models, resulting in enhanced efficiency and reliability of SR processes. Notably, ChatGPT exhibits exceptional performance in filtering and categorizing relevant articles, leading to significant time and effort savings. Our quantitative assessment reveals the following: (1) the overall accuracy of ChatGPT for article discarding and classification is 88%, and (2) the F-1 scores of ChatGPT for article discarding and classification are 91% and 88%, respectively, compared to expert assessments. However, we identify limitations in its suitability for article extraction. Overall, this research contributes valuable insights to the field of SR, empowering researchers to conduct more comprehensive and reliable reviews while advancing knowledge and decision-making across various domains.