Predictive Validity of Anesthesiologists’ Quality of Clinical Supervision and Nurse Anesthetists’ Work Habits Assessed by Their Associations With Operating Room Times

Abstract

BACKGROUND: At all Joint Commission-accredited hospitals, the anesthesia department chair must report quantitative assessments of anesthesiologists’ and nurse anesthetists’ (CRNAs’) clinical performance at least annually. Most metrics lack evidence of usefulness, cost-effectiveness, reliability, or validity. Earlier studies showed that anesthesiologists’ clinical supervision quality and CRNAs’ work habits have content, convergent, discriminant, and construct validity. We evaluated predictive validity by testing for (expected) small but statistically significant associations between higher quality of supervision (work habits) and reduced probabilities of cases taking longer than estimated. METHODS: Supervision quality of each anesthesiologist was evaluated daily by assigned trainees using the 9-item de Oliveira Filho scale. The work habits of each CRNA were evaluated daily by assigned anesthesiologists using a 6-item scale. Both are scored binary, 1 if all items are rated the maximum, 0 otherwise. From 40,718 supervision evaluations and 53,722 work habit evaluations over 8 fiscal years, 16 mixed-effects logistic regression models were estimated, with raters as fixed effects and ratees (anesthesiologists or CRNAs) as random effects. Empirical Bayes means in the logit scale were obtained for 561 anesthesiologist-years and 605 CRNA-years. The binary-dependent variable was whether the case took longer than estimated from the historical mean time for combinations of scheduled procedures and surgeons. From 264,060 cases, 8 mixed-effects logistic regression models were fitted, 1 per fiscal year, using ratees as random effects. Predictive validity was tested by pairing the 8 one-year analyses of clinical supervision, and the 8 one-year analyses of work habits, by ratee, with the 8 one-year analyses of whether OR time was longer than estimated. Bivariate errors in variable linear least squares linear regressions minimized total variances. RESULTS: Among anesthesiologists, 8.2% (46/561) had below-average supervision quality, and 17.7% (99/561), above-average. Among CRNAs, 6.3% (38/605) had below-average work habits, and 10.9% (66/605) above-average. Increases in the logits of the quality of clinical supervision were associated with decreases in the logits of the probabilities of cases taking longer than estimated, unitless slope = −0.0361 (SE, 0.0053), P < .00001. Increases in the logits of CRNAs’ work habits were associated with decreases in the logits of probabilities of cases taking longer than estimated, slope = −0.0238 (SE, 0.0054), P < .00001. CONCLUSIONS: Predictive validity was confirmed, providing further evidence for using supervision and work habits scales for ongoing professional practice evaluations. Specifically, OR times were briefer when anesthesiologists supervised residents more closely, and when CRNAs had better work habits.