Agreement and Reliability Analysis of Machine Learning Scaling in the Assessment of Acute Proximal Weakness by Experts and Non-Experts: Clinical Feasibility Study (Preprint)

Abstract

Assessing the symptoms of proximal weakness caused by neurological deficits requires expert knowledge and experienced neurologists. Recent advances in artificial intelligence and the Internet of Things have resulted in the development of automated systems that emulate physicians’ assessments.

This study provides an agreement and reliability analysis of using an automated scoring system to evaluate proximal weakness by experts and non-experts.

We collected 144 observations from acute stroke patients in a neurological intensive care unit to measure the symptom of proximal weakness of upper and lower limbs. A neurologist performed a gold standard assessment and two medical students performed identical tests as non-expert assessments for manual and machine learning-based scaling of Medical Research Council (MRC) proximal scores. The system collects signals from sensors attached on patients’ limbs and trains a machine learning assessment model using the hybrid approach of data-level and algorithm-level methods for the ordinal and imbalanced classification in multiple classes. For the agreement analysis, we investigated the percent agreement of MRC proximal scores and Bland-Altman plots of kinematic features between the expert- and non-expert scaling. In the reliability analysis, we analysed the intra-class correlation coefficients (ICCs) of kinematic features and Krippendorff’s alpha of the three observers’ scaling.

The mean percent agreement between the gold standard and the non-expert scaling was 0.542 for manual scaling and 0.708 for IoT-assisted machine learning scaling, with 30.63% enhancement. The ICCs of kinematic features measured using sensors ranged from 0.742 to 0.850, whereas the Krippendorff’s alpha of manual scaling for the three observers was 0.275. The Krippendorff’s alpha of machine learning scaling increased to 0.445, with 61.82% improvement.

Automated scaling using sensors and machine learning provided higher inter-rater agreement and reliability in assessing acute proximal weakness. The enhanced assessment supported by the proposed system can be utilized as a reliable assessment tool for non-experts in various emergent environments.