ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude

Abstract

Since ChatGPT-3.5 has been available to the public, the potentials and challenges regarding chatbot usage in education have been widely discussed. However, little evidence exists whether and for which purposes students even apply generative AI tools. The first main purpose of the present study was to develop and test scales that assess students’ (1) knowledge about ChatGPT, (2) actual ChatGPT usage and perceived value of use, and (3) attitude towards ChatGPT. Our second aim was to examine the intercorrelations between these scales, and to investigate differences (a) across five academic fields (i.e., human sciences, social sciences, teaching profession, health sciences, and law and economics) and (b) between stages of education (i.e., number of semesters). <i>N</i> = 693 students from various German universities participated in our online survey. Quality checks (Cronbach’s alpha, MacDonald’s omega, and confirmatory factor analyses) show satisfactory results for all scales. The scales all positively relate to each other, except for the knowledge and attitude scales. This means that more knowledge about ChatGPT is connected to a less favorable attitude regarding the generative AI tool. Lastly, MANOVA and subsequent Bonferroni corrected ANOVA tests show that ChatGPT is mostly used by law and economics students, and most frequently by students in the third year of higher education.