Crowd Annotations Can Approximate Clinical Autism Impressions from Short Home Videos with Privacy Protections

Abstract

AbstractArtificial Intelligence (A.I.) solutions are increasingly considered for telemedicine. For these methods to adapt to the field of behavioral pediatrics, serving children and their families in home settings, it will be crucial to ensure the privacy of the child and parent subjects in the videos. To address this challenge in A.I. for healthcare, we explore the potential for global image transformations to provide privacy while preserving behavioral annotation quality. Crowd workers have previously been shown to reliably annotate behavioral features in unstructured home videos, allowing machine learning classifiers to detect autism using the annotations as input. We evaluate this method with videos altered via pixelation, dense optical flow, and Gaussian blurring. On a balanced test set of 30 videos of children with autism and 30 neurotypical controls, we find that the visual privacy alterations do not drastically alter any individual behavioral annotation at the item level. The AUROC on the evaluation set was 90.0% +/- 7.5% for the unaltered condition, 85.0% +/- 9.0% for pixelation, 85.0% +/- 9.0% for optical flow, and 83.3% +/- 9.3% for blurring, demonstrating that an aggregation of small changes across multiple behavioral questions can collectively result in increased misdiagnosis rates. We also compare crowd answers against clinicians who provided the same annotations on the same videos and find that clinicians are more sensitive to autism-related symptoms. We also find that there is a linear correlation (r=0.75, p<0.0001) between the mean Clinical Global Impression (CGI) score provided by professional clinicians and the corresponding classifier score emitted by the logistic regression classifier with crowd inputs, indicating that the classifier’s output probability is a reliable estimate of clinical impression of autism from home videos. A significant correlation is maintained with privacy alterations, indicating that crowd annotations can approximate clinician-provided autism impression from home videos in a privacy-preserved manner.