Music teachers’ labeling accuracy and quality ratings of lesson plans by artificial intelligence (AI) and humans

Abstract

This study explored the potential of artificial intelligence (ChatGPT) to generate lesson plans for music classes that were indistinguishable from music lesson plans created by humans, with current music teachers as assessors. Fifty-six assessors made a total of 410 ratings across eight lesson plans, assigning a quality score to each lesson plan and labeling if they believed each lesson plan was created by a human or generated by AI. Despite the human-made lesson plans being rated higher in quality as a group ( p < .01, d = 0.44), assessors were unable to accurately label if a lesson plan was created by a human or generated by AI (55% accurate overall). Labeling accuracy was positively predicted by quality scores on human-made lesson plans and previous personal use of AI, while accuracy was negatively predicted by quality scores on AI-generated lesson plans and perception of how useful AI will be in the future. Open-ended responses from 42 teachers suggested assessors used three factors when making evaluations: specific details, evidence of classroom knowledge, and wording. Implications provide suggestions for how music teachers can use prompt engineering with a GPT model to create a virtual assistant or Intelligent Tutor System (ITS) for their classroom.