Automated identification and quantification of stereotypical movements from video recordings of children with ASD

Abstract

AbstractImportanceStereotypical motor movements (SMMs) are a form of restricted and repetitive behavior (RRB), which is a core symptom of Autism Spectrum Disorder (ASD). Current quantification of SMM severity is extremely limited, with studies relying on coarse and subjective caregiver reports or laborious manual annotation of short video recordings.ObjectiveTo demonstrate the utility of a new open-source AI algorithm that can analyze extensive video recordings of children and automatically identify segments with heterogeneous SMMs, thereby enabling their direct and objective quantification.Design, setting, and participantsThis retrospective cohort study analyzed video recordings from 319 behavioral assessments of 241 children with ASD, 1.4 to 8 years old, who participated in research at the Azrieli National Centre for Autism and Neurodevelopment Research in Israel. Behavioral assessments included cognitive, language, and autism diagnostic observation schedule, 2ndedition (ADOS-2) assessments.ExposuresEach assessment was recorded with 2-4 cameras, yielding 580 hours of video footage. We manually annotated 7,352 video segments containing heterogeneous SMMs performed by different children (21.14 hours of video).Main outcomes and measuresWe used a pose-estimation algorithm (OpenPose) to extract skeletal representations of all individuals in each video frame and trained an object-detection algorithm (YOLOv5) to identify and track the child in each movie. We then used the skeletal representation of the child to train an SMM recognition algorithm using a PoseConv3D model. We used data from 220 children for training and data from the remaining 21 children for testing.ResultsThe algorithm accurately detected 92.53% of manually annotated SMMs in our test data with 66.82% precision. Overall number and duration of algorithm identified SMMs per child were highly correlated with manually annotated number and duration of SMMs (r=0.8 and r=0.88, p<0.001 respectively).Conclusion and relevanceThese findings demonstrate the ability of the algorithm to capture a highly diverse range of SMMs and quantify them with high accuracy, enabling objective and direct estimation of SMM severity in individual children with ASD. We openly share the “ASDPose” dataset and “ASDMotion” algorithm for further use by the research community.Key PointsQuestionIs it possible to train a deep learning algorithm to accurately identify and quantify stereotypical motor movements (SMMs) in video recordings of children with autism?FindingsThe ASDMotion algorithm was trained and tested with the largest video dataset of ASD children curated to date, comprised of 319 behavioral assessment recordings from 241 ASD children. The algorithm successfully detected 92.53% of manually identified SMMs with 66.82% precision, achieving highly accurate quantification of SMMs per child that were strongly correlated (r≥0.8) with quantification by manual annotation.MeaningThis study demonstrates the utility of ASDMotion for objective and direct quantification of SMM severity in children with ASD, offering a new freely available, open-source algorithm and dataset that enable transformative basic and clinical ASD research.