Visualising candidate behaviour in computer-based testing: UsingClickMapsfor exploringClickStreamsin undergraduate and postgraduate medical examinations

Abstract

AbstractBackgroundThe rapid introduction of computer-based testing (CBT) in UK (United Kingdom) undergraduate and postgraduate medical education, mainly as a result of the COVID-19 pandemic, has generated large amounts of examination data, which we call theClickStream. As candidates navigate through exams, read questions, view images, choose answers and sometimes change answers and return, re-read, and make further changes, the multiple actions are recorded as a series of time-stamped clicks or keystrokes. Analysing that mass of data is far from simple, and here we describe the creation ofClickMaps, which allow examiners, educationalists and candidates to visualise behaviour in examinations.MethodsAs an example ofClickMaps, we describe data from a single examination lasting three hours, with 100 ‘best-of-five’ questions, which was one of two papers sat in 2021 by 508 candidates as a part of the MRCP(UK) Part 2 exam. TwoClickMapswere generated for each candidate. TheFull ClickMapallows the complete three-hours of the examination to be visualised, while theEarly ClickMap, shows in more detail how candidates responded during the first six minutes of presentation of each of the 100 questions in the exam.ResultsSince the primary purpose of this paper is expository, detailed descriptions and examples ofClickMapsfrom eleven candidates were chosen to illustrate different patterns of responding, both common and rare, and to show how straightforward areClickMapsto read and interpret.ConclusionsThe richness of the data inClickStreamsallows a wide range of practical and theoretical questions to be asked about how candidates behave in CBTs, which are considered in detail.ClickMapsmay also provide a useful method for providing detailed feedback to candidates who have taken CBTs, not only of their own behaviour but also for comparison with different strategies used by other candidates, and the possible benefits and problems of different approaches. In research terms, educationalists urgently need to understand how differences inClickMapsrelate to differences in student characteristics and overall educational performance.