The Black Box of Technological Outcome Measures: An Example in Duchenne Muscular Dystrophy

Abstract

Background: Outcome measures for non-ambulant Duchenne muscular dystrophy (DMD) patients are limited, with only the Performance of the Upper Limb (PUL) approved as endpoint for clinical trials. Objective: We assessed four outcome measures based on devices developed for the gaming industry, aiming to overcome disadvantages of observer-dependency and motivation. Methods: Twenty-two non-ambulant DMD patients (range 8.6–24.1 years) and 14 healthy controls (HC; range 9.5–25.4 years) were studied at baseline and 16 patients at 12 months using Leap Motion to quantify wrist/hand active range of motion (aROM) and a Kinect sensor for reached volume with Ability Captured Through Interactive Video Evaluation (ACTIVE), Functional Workspace (FWS) summed distance to seven upper extremity body points, and trunk compensation (KinectTC). PUL 2.0 was performed in patients only. A stepwise approach assessed quality control, construct validity, reliability, concurrent validity, longitudinal change and patient perception. Results: Leap Motion aROM distinguished patients and HCs for supination, radial deviation and wrist flexion (range p = 0.006 to <0.001). Reliability was low and the manufacturer’s hand model did not match the sensor’s depth images. ACTIVE differed between patients and HCs (p < 0.001), correlated with PUL (rho = 0.76), and decreased over time (p = 0.030) with a standardized response mean (SRM) of –0.61. It was appraised as fun on a 10-point numeric rating scale (median 9/10). PUL decreased over time (p < 0.001) with an SRM of –1.28, and was appraised as fun (median 7/10). FWS summed distance distinguished patients and HCs (p < 0.001), but reliability in patients was insufficient. KinectTC differed between patients and HCs (p < 0.01), but correlated insufficiently with PUL (rho = –0.69). Conclusions: Only ACTIVE qualified as potential outcome measure in non-ambulant DMD patients, although the SRM was below the commonly used threshold of 0.8. Lack of insight in technological constraints due to intellectual property and software updates made the technology behind these outcome measures a kind of black box that could jeopardize long-term use in clinical development.