Transforming Learning: Assessing the Efficacy of a Retrieval-Augmented Generation System as a Tutor for Introductory Psychology

Abstract

The introduction of Large Language Models (LLMs) have captured public imagination and represent a marked improvement in AI in Education (AIED) capabilities. But there is concern that student reliance on automated tools to complete written assignments may lead to a decline in learning. This study investigated whether participant use of LLMs to complete a writing assignment affected retention of learning content. Undergraduate participants ( N = 109) were randomly assigned to complete a writing assignment under one of three conditions: (1), with the assistance of a Retrieval-Augmented Generation (RAG)-based AI psychology tutor; (2) with the assistance of unmodified GPT-4 Turbo; (3) with no AI assistance. After completing the writing task, students completed a posttest quiz to assess their retention of learning material. The control condition had the lowest mean quiz score (M = 9.22, SD = 3.90), followed by the RAG AI tutor condition (M = 10.81, SD = 4.12), and unmodified GPT-4Turbo (M = 11.31, SD = 3.88), with significant differences between the AI tutor condition and the control condition (p = .036); and between the GPT-4 Turbo condition and control (p = .003); but not between the AI tutor and GPT-4 Turbo conditions (p = .283).