Construct Validity of Medical Clinical Competence Measures: A Multitrait-Multimethod Matrix Study Using Confirmatory Factor Analysis

Abstract

This study investigates the construct validity of three methods used to evaluate clinical competence in medicine: standardized test, supervisor performance ratings, and peer performance ratings. Three attributes of clinical competence are investigated: cognitive abilities, interpersonal skills, and professional qualities. Measures representing each attribute-method combination include: National Board of Medical Examiners (NBME) examination (standardized test of cognitive abilities); two scales derived from the California Psychological Inventory (CPI) (standardized test of interpersonal skills and professional qualities); and the three scales derived from the Resident Evaluation Form (REF) (peer and supervisor ratings of all three attributes). Scores for each attribute-method combination were obtained from a convenience sample of 166 resident physicians in three primary care specialties. These scores were cast into a multitrait-multimethod matrix design and analyzed using confirmatory factor analysis. Results suggest a lack of construct validity for the CPI and REF scales, moderate convergent validity for the NBME, and substantial method variance in the REF-derived ratings. Findings are discussed in terms of the implications for a theory of medical clinical competence, further research and development in clinical competence measurement, and current measurement practice in medical education.