LLMscreen: A Python Package for Systematic Review Screening of Scientific Texts Using Prompt Engineering

Abstract

Systematic reviews represent a cornerstone of evidence-based research, yet the process is labor-intensive and time-consuming, often requiring substantial human resources. The advent of Large Language Models (LLMs) offers a novel approach to streamlining systematic reviews, particularly in the title and abstract screening phase. This study introduces a new Python package built on LLMs to accelerate this process, evaluating its performance across three datasets using distinct prompt strategies: single-prompt, k-value setting, and zero-shot. The k-value setting approach emerged as the most effective, achieving a precision of 0.649 and reducing the average error rate to 0.4%, significantly lower than the 10.76% error rate typically observed among human reviewers. Moreover, this approach enabled the screening of 3,000 papers in under 8 minutes, at a cost of only $0.30—an over 250-fold improvement in time and 2,000-fold cost efficiency compared to traditional methods. These findings underscore the potential of LLMs to enhance the efficiency and accuracy of systematic reviews, though further research is needed to address challenges related to dataset variability and model transparency. Expanding the application of LLMs to other stages of systematic reviews, such as data extraction and synthesis, could further streamline the review process, making it more comprehensive and less burdensome for researchers.