Affiliation:
1. Boston College, Chestnut Hill, MA, USA
Abstract
Automated scoring of free drawings or images as responses has yet to be used in large-scale assessments of student achievement. In this study, we propose artificial neural networks to classify these types of graphical responses from a TIMSS 2019 item. We are comparing classification accuracy of convolutional and feed-forward approaches. Our results show that convolutional neural networks (CNNs) outperform feed-forward neural networks in both loss and accuracy. The CNN models classified up to 97.53% of the image responses into the appropriate scoring category, which is comparable to, if not more accurate, than typical human raters. These findings were further strengthened by the observation that the most accurate CNN models correctly classified some image responses that had been incorrectly scored by the human raters. As an additional innovation, we outline a method to select human-rated responses for the training sample based on an application of the expected response function derived from item response theory. This paper argues that CNN-based automated scoring of image responses is a highly accurate procedure that could potentially replace the workload and cost of second human raters for international large-scale assessments (ILSAs), while improving the validity and comparability of scoring complex constructed-response items.
Subject
Applied Mathematics,Applied Psychology,Developmental and Educational Psychology,Education
Reference43 articles.
1. Allaire J. J., Chollet F. (2021). keras: R interface to “keras” (R Package Version 2.6.0). https://CRAN.R-project.org/package=keras
2. Attali Y., Burstein J. (2006). Automated essay scoring with e-rater ® V.2. The Journal of Technology, Learning and Assessment, 4(3), 1-29. https://ejournals.bc.edu/index.php/jtla/article/view/1650
3. Rater Cognition: Implications for Validity
4. A Historical Survey of Research Regarding Constructed-Response Formats
5. Chollet F. (2018). Deep learning with Python. Manning Publications Co. https://www.manning.com/books/deep-learning-with-python
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献