Large-scale diagnostic assessment in first-year university students: pre- and transpandemic comparison

Abstract

AbstractCOVID-19 has disrupted higher education globally, and there is scarce information about the “learning loss” in university students throughout this crisis. The goal of the study was to compare scores in a large-scale knowledge diagnostic exam applied to students admitted to the university, before and during the pandemic. Research design was quasi-experimental with static group comparisons, taking advantage of the pandemic “natural experiment,” to assess knowledge in students admitted to the National Autonomous University of Mexico. Four student cohorts were analyzed: 2017 and 2018 (prepandemic, paper-and-pencil exams), 2020 and 2021 (transpandemic, online exams). The same instruments were applied in each pair of cohorts (2017–2021; 2018–2020) to decrease instrumentation threat. Propensity score matching was used to create balanced comparable groups. 35,584 matched students from each of the 2018 and 2020 cohorts were compared and 31,574 matched students from each of the 2017–2021 cohorts. Reliability and point biserial correlation coefficients were higher in the transpandemic online applications. Knowledge scores were 2.3 to 7.1% higher in the transpandemic assessments, Spanish scores in the 2018–2020 comparison were 1.3% lower, and English results in 2021 were 7.1% lower than in 2017. Before the pandemic, there was a 3.1% higher test performance in men; this gap decreased to 0.34% during the pandemic. There was no documented learning loss in this large student population, with an increase in knowledge in the pandemic cohorts. Some influence in scores due to the online testing modality cannot be ruled out. Longitudinal follow-up is required to continue evaluating the impact of the pandemic in learning.