The Accuracy of the Learning-Curve Cumulative Sum Method in Assessing Brachial Plexus Block Competency

Abstract

BACKGROUND: The learning-curve cumulative sum method (LC-CUSUM) and its risk-adjusted form (RA-LC-CUSUM) have been proposed as performance-monitoring methods to assess competency during the learning phase of procedural skills. However, scarce data exist about the method’s accuracy. This study aimed to compare the accuracy of LC-CUSUM forms using historical data consisting of sequences of successes and failures in brachial plexus blocks (BPBs) performed by anesthesia residents. METHODS: Using historical data from 1713 BPB performed by 32 anesthesia residents, individual learning curves were constructed using the LC-CUSUM and RA-LC-CUSUM methods. A multilevel logistic regression model predicted the procedure-specific risk of failure incorporated in the RA-LC-CUSUM calculations. Competency was defined as a maximum 15% cumulative failure rate and was used as the reference for determining the accuracy of both methods. RESULTS: According to the LC-CUSUM method, 22 residents (84.61%) attained competency after a median of 18.5 blocks (interquartile range [IQR], 14–23), while the RA-LC-CUSUM assigned competency to 20 residents (76.92%) after a median of 17.5 blocks (IQR, 14–25, P = .001). The median failure rate at reaching competency was 6.5% (4%–9.75%) under the LC-CUSUM and 6.5% (4%–9%) for the RA-LC-CUSUM method (P = .37). The sensitivity of the LC-CUSUM (85%; 95% confidence interval [CI], 71%–98%) was similar to the RA-LC-CUSUM method (77%; 95% CI, 61%–93%; P = .15). Identical specificity values were found for both methods (67%; 95% CI, 29%–100%, P = 1). CONCLUSIONS: The LC-CUSUM and RA-LC-CUSUM methods were associated with substantial false-positive and false-negative rates. Also, small lower limits for the 95% CIs around the accuracy measures were observed, indicating that the methods may be inaccurate for high-stakes decisions about resident competency at BPBs.