Interrater Agreement on National Institutes of Health Stroke Scale Between Paramedics and Stroke Physicians: Validation Study for the Digital Training Model in the Paramedic Norwegian Acute Stroke Prehospital Project

Abstract

Background Time spent in the prehospital phase of acute stroke care is multifactorial and has an effect on the possibilities for acute treatment. Communication between paramedics and the in-hospital stroke team directly affects time to treatment. A mutual stroke scale such as the National Institutes of Health Stroke Scale (NIHSS) may improve communication quality. The Paramedic Norwegian Acute Stroke Prehospital Project (ParaNASPP) was a stepped-wedge, randomized trial of stroke screening using NIHSS in the ambulance where the intervention was training paramedics in stroke and the NIHSS, with the use of NIHSS made into a mobile app to guide the examination and facilitate communication with the in-hospital stroke team. Objective The aim of this study was to validate the digital training model from the ParaNASPP clinical trial. Methods In total, 24 paramedics were recruited from Oslo University Hospital in Norway to complete the ParaNASPP training model; 20 exclusive videos with predefined NIHSS scores were recorded; and 4 stroke physicians from Oslo University Hospital were included for reference. Bland-Altman plots with 95% limits of agreement (LoA) were calculated—first comparing paramedics and stroke physicians to the predefined scores and then with each other. The predefined LoA were set to 3 points. To align with clinical practice, NIHSS scores were also dichotomized into 2 categories: from 0-5 (minor stroke) or ≥6 (moderate and major stroke), and agreement was calculated using Cohen κ. Results The videos (n=20) had a median (range) NIHSS score of 7 (0-31). The paramedics’ scores were slightly higher than the predefined scores with a mean difference of –0.38 and the LoA ranging from –4.04 to 3.29. The paramedics scored higher than the stroke physicians with a mean difference of –0.39, with the LoA ranging from –4.58 to 3.80. When the NIHSS scores were dichotomized, Cohen κ was 0.89 between the predefined scores and paramedics, 0.92 between the predefined scores and stroke physicians, and 0.81 between the paramedics and stroke physicians, all indicating very good agreement. Conclusions The paramedics scored higher than both the predefined scores and stroke physicians’ scores, hence the predefined LoA were not met. However, the width of the LoA was smaller than seen when experienced neurologists are compared. When the NIHSS scores were dichotomized, the paramedics achieved very good agreement with both the predefined scores and stroke physicians’ scores. This study demonstrates the possibilities for the transfer of clinical competence in digital simulation training.