Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery

Abstract

OBJECTIVE The authors previously demonstrated the use of a validated Objective Structured Assessment of Technical Skills (OSATS) tool for evaluating residents' operative skills in pediatric neurosurgery. However, no benchmarks have been established for specific pediatric procedures despite an increased need for meaningful assessments that can either allow for early intervention for underperforming trainees or allow for proficient residents to progress to conducting operations independently with more passive supervision. This validated methodology and tool for assessment of operative skills for common pediatric neurosurgical procedures—external ventricular drain (EVD) placement and shunt surgery— was applied to establish its procedure-based feasibility and reliability, and to document the effect of repetition on achieving surgical skill proficiency in pediatric EVD placement and shunt surgery. METHODS A procedure-based technical skills assessment for EVD placements and shunt surgeries in pediatric neurosurgery was established through the use of task analysis. The authors enrolled all residents from 3 training programs (Baylor College of Medicine, Houston Methodist Hospital, and University of Texas–Medical Branch) who rotated through pediatric neurosurgery at Texas Children's Hospital over a 26-month period. For each EVD placement or shunt procedure performed with a resident, the faculty and resident (for self-assessment) completed an evaluation form (OSATS) based on a 5-point Likert scale with 7 categories. Data forms were then grouped according to faculty versus resident (self) assessment, length of pediatric neurosurgery rotation, postgraduate year level, and date of evaluation (“beginning of rotation,” within 1 month of start date; “end of rotation,” within 1 month of completion date; or “middle of rotation”). Descriptive statistical analyses were performed with the commercially available SPSS statistical software package. A p value < 0.05 was considered statistically significant. RESULTS Five attending evaluators (including 2 fellows who acted as attending surgeons) completed 260 evaluations. Twenty house staff completed 269 evaluations for self-assessment. Evaluations were completed in 562 EVD and shunt procedures before the surgeons left the operating room. There were statistically significant differences (p < 0.05) between overall attending (mean 4.3) and junior resident (self; mean 3.6) assessments, and between overall attending (mean 4.8) and senior resident (self; mean 4.6) assessment scores on general performance and technical skills. The learning curves produced for the residents demonstrate a stereotypical U- or V-shaped curve for acquiring skills, with a significant improvement in overall scores at the end of the rotation compared with the beginning. The improvement for junior residents (Δ score = 0.5; p = 0.002) was larger than for senior residents (Δ score = 0.2; p = 0.018). CONCLUSIONS The OSATS is an effective assessment tool as part of a comprehensive evaluation of neurosurgery residents' performance for specific pediatric procedures. The authors observed a U-shaped learning curve, contradicting the idea that developing one's surgical technique and learning a procedure represents a monotonic, cumulative process of repetitions and improvement.