Cross-sectional evaluation of an asynchronous Multiple Mini Interview (MMI) in selection to health professions training programmes with ten principles for fairness built-in

Abstract

AbstractObjectivesEnsuring equity, inclusivity, and diversity in health professions selection is an ethical and practical imperative. We have built the first known online asynchronous Multiple Mini-Interview (MMI).We aimed to explore psychometric properties for all users with sub-group analysis by key characteristics, acceptability, and usability.Design, setting, participantsCross-discipline multi-method evaluation with applicants to Nursing, Midwifery and Paramedic Science under-graduate programmes from one UK university (2021/2022).Primary, secondary outcome measuresPsychometric properties (internal consistency, construct validity, dimensionality) were assessed using Cronbach’s alpha (α), parallel analysis (PA), Schmid-Leiman transformation and ordinal confirmatory factor analysis (CFA). Usability and acceptability were evaluated using descriptive statistics and conventional content analysis.MethodsThe system was configured in a seven question four-minute MMI. Applicants’ video-recorded their answers which were later assessed by interviewers and scores summed. Applicants and interviewers completed online evaluation questionnaires.ResultsPerformance data from 712 applicants determined good-excellent reliability for the asynchronous MMI assessment (mean α 0.72) with similar results across sub-groups (gender, age, disability/support needs, UK/non-UK). Parallel analysis and factor analysis results suggested that there were seven factors relating to the MMI questions with an underlying general factor that explained the variance in observed candidate responses. A confirmatory factor analysis testing a seven-factor hierarchical model showed an excellent fit to the data (Confirmatory Fit Index =0.99), Tucker Lewis Index =0.99, RMSE=0.034).Applicants (n=210) viewed the flexibility, relaxed environment, and cost savings advantageous. Interviewers (n=65) reported the system intuitive, flexible with >70% time saved compared to face-to-face interviews. Reduced personal communication was cited as the principle disadvantage.ConclusionsOur findings suggest that the asynchronous MMI is reliable, time-efficient, fair, and acceptable. In the absence of any known precedent, these internationally applicable, cross discipline insights inform the future configuration of online interviews where building-in principles for fairness are relatively straight forward to implement.Study strengths and limitationsThe theoretical approach aligned with an iterative process necessary to design a new technology to reduce bias.The large sample enabled us to assess psychometric properties with sub-group analysis for the first time in this context.The study provides perspectives from one large site; a necessary step to inform a planned international multi-site evaluation.The multi-method design provided insights necessary to embed fairness into online selection approaches in the absence of best practice guidance.