Development of the Cognitive Load Inventory for Surgical Skills: preliminary validity evidence

Abstract

Abstract Purpose The operating room requires simultaneous processing of considerable information. Trainee learning and performance in this environment are restricted by working memory. Cognitive learning theory, which describes three types of cognitive load (CL), suggests that learning is maximized when intrinsic load (IL) is matched to experience, extraneous load (EL) is minimized, and germane load (GL) is optimized with schemas. The ability to measure CL is critical to maximize learning. The Cognitive Load Inventory for Colonoscopy (CLIC) measures the three types of CL during colonoscopy training. We adapted the CLIC for surgical skills education and aimed to gather validity evidence for this tool. Methods We initially developed a 21-item, 5-point Likert instrument, the Cognitive Load Inventory for Surgical Skills (CLISS), through a multi-step process. To support content validity, we revised CLIC content to reflect surgical skills, obtaining feedback from CLIC creators to ensure adjustments adequately retained IL, EL, and GL concepts. To examine the response process, we conducted cognitive interviews with surgical residents to assess wording and clarity and issued the survey in different settings (simulation exercises, operating room cases), relative to different tasks, and with different respondents (residents, fellows, faculty). To assess internal structure, we examined internal consistency using Cronbach's alpha and conducted confirmatory factor analysis (CFA) to confirm that relationships among the items were as hypothesized. We evaluated adequacy of the 3-factor model with three model fit indices, including Bentler’s Comparative Fit Index (CFI), the Tucker–Lewis Index (TLI), and root mean square error of approximation (RMSEA). A sensitivity analysis was performed eliminating items with factor loading estimates that (1) are less than 0.40, (2) are not statistically significant, or (3) load onto 2 or more factors. Results The CLISS was distributed in 7 clinical and simulated procedural settings to 138 participants and yielded 100 responses (72% response rate). Of the respondents, 99 (99%) completed the entire CLISS. No respondents raised content questions. Each item garnered a range of responses. All 3 types of cognitive load had a Cronbach’s alpha above 0.7. However, several individual items were found not to correlate with other items within that load. CFA revealed a range of loadings for each factor. Model fit indices were outside the conventional cut-offs (CFI = 0.627, TLI = 0.579, RMSEA = 0.124), suggesting consensus that the initial 3-factor, 21-item model was not a good fit. Sensitivity analysis yielded a revised 11-item tool, which demonstrated improved Cronbach’s alpha for IL and GL and improved model indices (CFI = 0.940, TLI = 0.920, RMSEA = 0.076). An additional item was removed to further support content validity. Conclusions The CLISS can be a practical tool for administration in the skills lab and clinical settings, particularly if administered immediately after the session or requested individually. While promising, further study is needed to establish additional validity evidence for the CLISS.