Development and evaluation of a serious game application to engage university students in critical thinking: a mixed-method study (Preprint)

Abstract

Serious games can affect motivation and learning. However, little is known about users’ experiences through the process from design and development to implementation and evaluation of a serious game application.

The aim of this study was to document how user insights can inform the concept and development of a serious game application in critical thinking, in addition to gather user experiences with the game application.

This was a mixed-methods study in four successive phases with both qualitative and quantitative data collected in the period from 2020 to 2022. Qualitative data on design and development was obtained from four focus group (FG) discussions, and qualitative evaluation data was obtained from one FG interview and three open-ended questions in the game application. The quantitative data originates from user statistics. The qualitative data were analyzed thematically, and user data were analyzed using non-parametric tests.

The first FG discussion revealed that the students’ (3 participants) assessment of whether a claim was reliable or not was limited to performing Google searches, when faced with an ad for a health intervention. Based on the acquired knowledge of the target group, the game’s prerequisites, and the technical possibilities, a pilot of the game was created, and reviewed question by question in three separate FG discussions (6 participants). After adjustments, the game was advertised at OsloMet, and 193 students tested the game. A correlation (r =0.77, P <.001) was found between number of replays and total points achieved in the game. There was no demonstrable difference (P=.073) between the total scores of students from different faculties. Seventy (36.3 %) students answered the evaluation questions in the game. They used words like “fun” and “educational” about the experiences with the game, and words like “motivating” and “engaging” related to the learning experience. The design was described as “varied” and “user-friendly”. Suggested improvements included adding references, more games and modules, more difficult questions, and an introductory text explaining the game. The results from the evaluation FG interview (4 participants) corresponded to a large extent with the results in the open-ended questions in the game.

Users can be successfully involved in all stages of the development and evaluation of a serious game that aims to engage students to think critically about health claims. Our approach can inform how such games can be designed and implemented in the development of other serious games where critical thinking about claims is important.