Validation of the Instructional Materials Motivation Survey among Medicine Resident Physicians

Abstract

Purpose The Instructional Materials Motivation Survey (IMMS) was developed to measure motivational characteristics of a learning activity, building on Keller's Attention, Relevance, Confidence, Satisfaction (ARCS) motivation model. We aimed to validate IMMS scores using validity evidence of internal structure and relations with other variables. Methods Participants were internal medicine and family medicine residents who completed the IMMS following an online module on outpatient medicine, from 2005 to 2009. We used confirmatory factor analysis (CFA) to examine model fit using half the data (split-sample approach). Finding suboptimal fit, we conducted exploratory factor analysis (EFA) and developed a revised instrument. We evaluated this instrument with CFA using the remaining data. Associations were evaluated between IMMS scores and knowledge and other measures of motivation (Motivated Strategies for Learning Questionnaire, MSLQ). All analyses accounted for repeated measures on subjects. Results There were 242 participants. Although internal consistency reliabilities were good (Cronbach alpha ≥0.70), CFA of the original 36-item, 4-domain instrument revealed poor model fit for data sample 1. EFA found that reverse-scored items clustered strongly together. Further EFA using data sample 1, followed by CFA using data sample 2, found good fit for a 13-item, 4-domain model that omitted reverse-scored items (standardized root mean square residual 0.045, root mean square error of approximation 0.066, comparative fit index 0.96). Linear regression confirmed positive, statistically significant associations for most hypothesized relationships, including IMMS total with knowledge (r=0.19) and MSLQ total (r=0.53; both p<.001). Examination of reverse-scored items suggested participant inattention but not acquiescence. Conclusions IMMS scores show good reliability and relations with other variables. However, the hypothesized and empirical factor structures do not align, and reverse-scored items show particularly poor fit. A 13-item, 4-domain scale omitting reverse-scored items showed good model fit.