Development and Validation of a Vocabulary Measure in the Mobile Toolbox

Abstract

Abstract Objective We describe the development of a new computer adaptive vocabulary test, Mobile Toolbox (MTB) Word Meaning, and validity evidence from 3 studies. Method Word Meaning was designed to be a multiple-choice synonym test optimized for self-administration on a personal smartphone. The items were first calibrated online in a sample of 7,525 participants to create the computer-adaptive test algorithm for the Word Meaning measure within the MTB app. In Study 1, 92 participants self-administered Word Meaning on study-provided smartphones in the lab and were administered external measures by trained examiners. In Study 2, 1,021 participants completed the external measures in the lab and Word Meaning was self-administered remotely on their personal smartphones. In Study 3, 141 participants self-administered Word Meaning remotely twice with a 2-week delay on personal iPhones. Results The final bank included 1363 items. Internal consistency was adequate to good across samples (ρxx = 0.78 to 0.81, p < .001). Test–retest reliability was good (ICC = 0.65, p < .001), and the mean theta score was not significantly different upon the second administration. Correlations were moderate to large with measures of similar constructs (ρ = 0.67–0.75, p < .001) and non-significant with measures of dissimilar constructs. Scores demonstrated small to moderate correlations with age (ρ = 0.35 to 0.45, p < .001) and education (ρ = 0.26, p < .001). Conclusion The MTB Word Meaning measure demonstrated evidence of reliability and validity in three samples. Further validation studies in clinical samples are necessary.