Affiliation:
1. University of Illinois Urbana–Champaign, USA
2. Georgia State University, USA
Abstract
Nonverbal behavior can impact language proficiency scores in speaking tests, but there is little empirical information of the size or consistency of its effects or whether language proficiency may be a moderating variable. In this study, 100 novice raters watched and scored 30 recordings of test takers taking an international, high stakes proficiency test. The speech samples were each 2 minutes long and ranged in proficiency levels. The raters scored each sample on fluency, vocabulary, grammar, and comprehensibility using 7-point semantic differential scales. Nonverbal behavior was extracted using an automated machine learning software called iMotions, and data was analyzed with ordinal mixed effects regression. Results showed that attentional variance predicted fluency, vocabulary, and grammar scores, but only when accounting for proficiency. Higher standard deviations of attention corresponded with lower scores for the lower-proficiency group, but not the mid/higher-proficiency group. Comprehensibility scores were only predicted by mean valence when proficiency was an interaction term. Higher mean valence, or positive emotional behavior, corresponded with higher scores in the lower-proficiency group, but not the mid/higher-proficiency group. Effect sizes for these predictors were quite small, with small amounts of variance explained. These results have implications for construct representation and test fairness.
Funder
British Council
The International Research Foundation
Duolingo
Reference53 articles.
1. Affectiva. (2022). Recent debates regarding facial expression technologies. https://2372627.fs1.hubspotusercontent-na1.net/hubfs/2372627/Recent%20debates%20regarding%20Facial%20Expression%20technologies_Affectiva.pdf
2. Affectiva. (n.d.). Affectiva media analytics. https://go.affectiva.com/affdex-for-market-research
3. Reliability and validity of machine vision for the assessment of facial expressions
4. Assessing Proportionality in the Proportional Odds Model for Ordinal Logistic Regression
5. Verbal and Nonverbal Communication: Distinguishing Symbolic, Spontaneous, and Pseudo-Spontaneous Nonverbal Behavior