Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery

Abstract

OBJECT Currently there is no standardized tool for assessment of neurosurgical resident performance in the operating room. In light of enhanced requirements issued by the Accreditation Council for Graduate Medical Education’s Milestone Project and the Matrix Curriculum Project from the Society of Neurological Surgeons, the implementation of such a tool seems essential for objective evaluation of resident competence. Beyond compliance with governing body guidelines, objective assessment tools may be useful to direct early intervention for trainees performing below the level of their peers so that they may be given more hands-on teaching, while strong residents can be encouraged by faculty members to progress to conducting operations more independently with passive supervision. The aims of this study were to implement a validated assessment tool for evaluation of operative skills in pediatric neurosurgery and determine its feasibility and reliability. METHODS All neurosurgery residents completing their pediatric rotation over a 6-month period from January 1, 2014, to June 30, 2014, at the authors’ institution were enrolled in this study. For each procedure, residents were evaluated by means of a form, with one copy being completed by the resident and a separate copy being completed by the attending surgeon. The evaluation form was based on the validated Objective Structured Assessment of Technical Skills for Surgery (OSATS) and used a 5-point Likert-type scale with 7 categories: respect for tissue; time and motion; instrument handling; knowledge of instruments; flow of operation; use of assistants; and knowledge of specific procedure. Data were then stratified by faculty versus resident (self-) assessment; postgraduate year level; and difficulty of procedure. Descriptive statistics (means and SDs) were calculated, and the results were compared using the Wilcoxon signed-rank test and Student t-test. A p value < 0.05 was considered statistically significant. RESULTS Six faculty members, 1 fellow, and 8 residents completed evaluations for 299 procedures, including 32 ventriculoperitoneal (VP) shunt revisions, 23 VP shunt placements, 19 endoscopic third ventriculostomies, and 18 craniotomies for tumor resection. There was no significant difference between faculty and resident self-assessment scores overall or in any of the 7 domains scores for each of the involved residents. On self-assessment, senior residents scored themselves significantly higher (p < 0.02) than junior residents overall and in all domains except for “time and motion.” Faculty members scored senior residents significantly higher than junior residents only for the “knowledge of instruments” domain (p = 0.05). When procedure difficulty was considered, senior residents’ scores from faculty members were significantly higher (p = 0.04) than the scores given to junior residents for expert procedures only. Senior residents’ self-evaluation scores were significantly higher than those of junior residents for both expert (p = 0.03) and novice (p = 0.006) procedures. CONCLUSIONS OSATS is a feasible and reliable assessment tool for the comprehensive evaluation of neurosurgery resident performance in the operating room. The authors plan to use this tool to assess resident operative skill development and to improve direct resident feedback.