Inter- and Intrarater Reliability of the Gap–Kalamazoo Communication Skills Assessment Form Among Occupational Therapy Interns

Abstract

Abstract Importance: Effective communication skills (CS) are essential for occupational therapists. The Gap–Kalamazoo Communication Skills Assessment Form (GKCSAF) is a standard tool for assessing the CS of medical residents. However, the interrater reliability for the nine CS domain scores ranges from poor to good. The intrarater reliability remains unclear. Objective: To examine the inter- and intrarater reliability of the GKCSAF’s nine domain scores and total score among occupational therapy interns. Design: Repeated assessments with the GKCSAF. Setting: Medical center psychiatry department. Participants: Twenty-five interns and 49 clients with mental illness, recruited from August 2020 to December 2021. Outcomes and Measures: The transcripts of 50 evaluation interviews between clients and interns were used. Three independent raters assessed each transcript twice, at least 3 mo apart. Results: The GKCSAF demonstrated poor interrater reliability for the nine domain scores (weighted κ = .08–.30) and the total score (intraclass correlation coefficient [ICC] = .22, 95% confidence interval [CI] [.10, .35]). The GKCSAF showed poor to intermediate intrarater reliability for the nine domain scores (weighted κ = .27–.73) and fair reliability for the total score (ICC = .69, 95% CI [.60, .77]). Conclusions and Relevance: The GKCSAF demonstrates poor interrater reliability and poor to intermediate intrarater reliability for the nine domain scores. However, it demonstrates fair intrarater reliability in assessing the overall CS performance of occupational therapy interns. Significant variations were observed when different raters assessed the same interns’ CS, indicating inconsistencies in ratings. Consequently, it is advisable to conservatively interpret the CS ratings obtained with the GKCSAF. Plain-Language Summary: It is essential for occupational therapists to effectively communicate with clients. The Gap–Kalamazoo Communication Skills Assessment Form (GKCSAF) is a standard tool that is used to assess the communication skills of medical residents. The study authors used the GKCSAF with occupational therapy interns in a medical center psychiatry department to assess how effectively they interviewed clients with mental illness. This study aids occupational therapy personnel in the interpretation of GKCSAF results. The study findings also highlight the importance of developing reliable and standardized measures to assess communications skills in the field of occupational therapy.