Abstract
AbstractUnfavorable test-taking behaviors, such as speededness and disengagement, have long been a validity concern for large-scale low-stakes assessments. Understanding the presence and extent of such behaviors is important for ensuring the validity of inferences based on test scores. This study examined test-taking behaviors using item response time (RT), a process data-derived variable from the TIMSS 2019 database. Analyses compared the United States to three other countries (England, Singapore, and the United Arab Emirates) that administered the digital version of TIMSS (eTIMSS) 2019 in English at grade 8. Test-taking behaviors were identified within each country and compared within and across countries. Specifically, to identify distinct types of test-taking behaviors, mixture modeling was employed on RT and item scores from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment. The results indicated that each country had several latent classes of students with different pacing trajectories and performance. The test-taking behaviors of these latent classes were labeled as Steady; Disengaged or Very disengaged; Speeded or Very speeded; and Efficient and high-performing. Most of the students in each country had a Steady pace (medium to high sum score; steady RT throughout the test): 71% in England, 74% in both Singapore and the United Arab Emirates, and 84% in the United States. Disengaged or Very disengaged students (low sum score; short RT) were identified in each country but were more prevalent in England and the United Arab Emirates (above 20% in both) than in the United States and Singapore (both below 10%). The study also revealed small percentages of Speeded or Very speeded students (low to medium sum score; long RT at first but very short RT toward the end) in England, the United Arab Emirates, and the United States (1%, 5%, and 6%, respectively) but not in Singapore. A unique class of Efficient and high-performing students (high sum score; short RT) was identified only in Singapore (24%). This study demonstrated that mixture modeling is a useful technique for identifying distinct test-taking behaviors and highlighted the presence and extent of unfavorable test-taking behaviors within each selected country using data from Booklet 1, Part 1, of the eTIMSS 2019 eighth-grade assessment.
Funder
National Center for Education Statistics (NCES) in the U.S.
Publisher
Springer Science and Business Media LLC
Reference33 articles.
1. Asparouhov, T., & Muthén, B. (2006). Multilevel modeling of complex survey data. In Proceedings of the joint statistical meeting in Seattle (pp. 2718–2726).
2. Bolt, D. M., Cohen, A. S., & Wollack, J. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331–348.
3. Celeux, G., & Soromenho, G. (1996). An entropy criterion for assessing the number of clusters in a mixture model. Journal of Classification, 13(2), 195–212.
4. Clark, S. L. (2010). Mixture modeling with behavioral data Doctoral dissertation. California: University of California.
5. Debeer, D., Bucholz, J., Hartig, J., & Janssen, R. (2014). Student, school, and country differences in sustained test-taking effort in the 2009 PISA Reading Assessment. Journal of Educational and Behavioral Statistics, 39, 502–523.