Interrater agreement between student and teacher assessments of endotracheal intubation skills in a self-directed simulation learning environment

Abstract

Abstract Background: Knowledge assessment is an important part of the learning process to confirm competencies in acquired medical knowledge. Objective: This study aimed to compare the assessments of endotracheal intubation skills using the HybridLab® methodology between students and teachers in terms of interobserver reliability. Methods: Reliability analysis was performed with observational data (data are reported according to STROBE guidelines). The study was conducted in two countries, the Lithuanian University of Health Science (LUHS) and Pennsylvania State University (PSU) in the US, between 1 January and 30 June 2020. A total of 92 students (60 from LUHS and 32 from PSU) were trained in endotracheal intubation using an algorithm-driven hybrid learning method. At the end of the training session, the participants had to complete the evaluation scenario, which was assessed by one of the students and evaluated remotely by a single teacher. The student assessment of the endotracheal intubation procedure was compared with the teacher’s assessment using correlation and estimation of the intraclass correlation coefficient. Results: Overall, the medians of the student and teacher assessments were both 100% (0%). Spearman’s correlation coefficient between the student and teacher assessments was 0.879 (p=0.001). The intraclass correlation coefficient used for interobserver variations between the students and teachers was 0.883 (95% confidence interval from 0.824 to 0.923). Conclusions: The algorithm-driven hybrid learning method allows students to reliably assess endotracheal intubation skills to a level comparable with that of the teacher’s evaluation.