The Covariance Decomposition of the Probability Score and Its Use in Evaluating Prognostic Estimates

Abstract

The probability score (PS) or Brier score has been used in a large number of studies in which physician judgment performance was assessed. However, the covariance decom position of the PS has not previously been used to evaluate medical judgment. The authors introduce the technique and demonstrate it by analyzing prognostic estimates of three groups: physicians, their patients, and the patients' decision-making surrogates. The major com ponents of the covariance decomposition—bias, slope, and scatter—are displayed in co variance graphs for each of the three groups. The decomposition reveals that whereas the physicians have the best overall estimation performance, their bias and their scatter are not always superior to those of the other two groups. This is primarily due to two factors. First, the physicians' prognostic estimates are pessimistic. Second, the patients place the large majority of their estimates in the most optimistic category, thereby achieving low scatter. The authors suggest that the calculational simplicity of this decomposition, its informative- ness, and the intuitive nature of its components make it a useful tool with which to analyze medical judgment. Key words: covariance decomposition; calibration; prognostic estimation; surrogate; probability score. (Med Decis Making 1995;15:120-131)