Large-scale capture of hidden fluorescent labels for training generalizable markerless motion capture models

Abstract

AbstractRecent advances in deep learning-based markerless pose estimation have dramatically improved the scale and ease with which body landmarks can be tracked in studies of animal behavior. However, pose estimation for animals in a laboratory setting still faces some specific challenges. Researchers typically need to manually generate new training data for each experimental setup and visual environment, limiting the generalizability of this approach. With each network being trained from scratch, different investigators track distinct anatomical landmarks and analyze the resulting kinematic data in idiosyncratic ways. Moreover, much of the movement data is discarded: only a few sparse landmarks are typically labeled, due to the inherent scale and accuracy limits of manual annotation. To address these issues, we developed an approach, which we term GlowTrack, for generating large training datasets that overcome the relatively modest limits of manual labeling, enabling deep learning models that generalize across experimental contexts. The key innovations are: a) an automated, high-throughput approach for generating hidden labels free of human error using fluorescent markers; b) a multi-camera, multi-light setup for generating large amounts of training data under diverse visual conditions; and c) a technique for massively parallel tracking of hundreds of landmarks simultaneously using computer vision feature matching algorithms, providing dense coverage for kinematic analysis at a resolution not currently available. These advances yield versatile deep learning models that are trained at scale, laying the foundation for standardized behavioral pipelines and more complete scrutiny of animal movements.