Development, validation and initial outcomes of a questionnaire to examine human factors in postgraduate surgical objective structured clinical examinations

Abstract

Abstract Background Human factors including stress, repetition, burnout and fatigue are associated with possible sources of error. Objective structured clinical examinations (OSCEs), where examiners concentrate for long periods, would benefit from a human factors approach to see whether these factors affect consistency of examiner behaviour, attitude and marking. Little has been published for OSCEs, in part due to the lack of a validated tool for collecting data in this setting. Methods A 46-item questionnaire was developed based on the Human Factors Analysis and Classification System (HFACS) domains and completed by examiners in the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination. To refine the questionnaire, an initial analysis focused on response patterns of each item. Cronbach's α was used to assess internal consistency, and a factor analysis was performed to uncover different domains emerging from the data. Results A total of 108 examiners completed the questionnaire (90·0 per cent response rate). The questionnaire, refined to 38 items based on an initial analysis of response patterns, showed good reliability for internal consistency (Cronbach's α = 0·76) and test–retest reliability (r = 0·85, n = 48, P < 0·001). Four factors had a close themed resemblance to the original HFACS domains, but were associated with different items, suggesting that the four human-factor domains might be linked to different behaviours and attitudes in an examination setting. Analyses according to sex, professional background and experience highlighted additional stress levels in examiners from one of the surgical Royal Colleges (P <0·001), matching evidence from the situation in that College at the time of this study. Conclusion The recognition and further investigation of human factors in OSCEs is needed to improve examiner experience and behaviour in order to influence delivery, candidate experience and quality assurance of these examinations.