A comparative study of AI-human-made and human-made test forms for a university TESOL theory course-Reference-Cited by-同舟云学术

A comparative study of AI-human-made and human-made test forms for a university TESOL theory course

Published:2024-06-07 Issue:1 Volume:14 Page:
ISSN:2229-0443
Container-title:Language Testing in Asia
language:en
Short-container-title:Lang Test Asia

Author:

O Kyung-Mi^ORCID

Abstract

AbstractThis study examines the efficacy of artificial intelligence (AI) in creating parallel test items compared to human-made ones. Two test forms were developed: one consisting of 20 existing human-made items and another with 20 new items generated with ChatGPT assistance. Expert reviews confirmed the content parallelism of the two test forms. Forty-three university students then completed the 40 test items presented randomly from both forms on a final test. Statistical analyses of student performance indicated comparability between the AI-human-made and human-made test forms. Despite limitations such as sample size and reliance on classical test theory (CTT), the findings suggest ChatGPT’s potential to assist teachers in test item creation, reducing workload and saving time. These results highlight ChatGPT’s value in educational assessment and emphasize the need for further research and development in this area.

Funder

Dongduk Women`s University

Publisher

Springer Science and Business Media LLC

Link

https://link.springer.com/content/pdf/10.1186/s40468-024-00291-3.pdf

Reference38 articles.

1. Abida, F. I. N., Kuswardani, R., Purwati, O., Rosyid, A., & Minarti, E. (2023). Assessing language proficiency through AI chatbot-based evaluation. In Proceedings of the International Conference on Islamic Civilization and Humanities (Vol. 1, pp. 138–145). Retrieved March 9, 2024 from https://proceedings.uinsby.ac.id/index.php/iconfahum/article/view/1230.

2. Adadan, E., & Savasci, F. (2011). An analysis of 16–17-year-old students’ understanding of solution chemistry concepts using a two-tier diagnostic instrument. International Journal of Science Education, 34(4), 513–544. https://doi.org/10.1080/09500693.2011.636084.

3. Aryadoust, V., Zakaria, A., & Jia, Y. (2024). Investigating the affordances of OpenAI’s large language model in developing listening assessments. Computers and Education: Artificial Intelligence, 6(2024), 100204. https://doi.org/10.1016/j.caeai.2024.100204.

4. Ayanwale, M., Chere-Masopha, J., & Morena, M. C. (2022). The classical test or item response measurement theory: the status of the framework at the examination council of Lesotho. International Journal of Learning, Teaching and Educational Research,21(8), 384–406. https://www.ijlter.org/index.php/ijlter/article/view/5676.

5. Baker, T., Smith, L., & Anissa, N. (2019). Educ-AI-tion rebooted? Exploring the future of artificial intelligence in schools and colleges. Nesta Foundation. https://media.nesta.org.uk/documents/Future_of_AI_and_education_v5_WEB.pdf.