DAMM for the detection and tracking of multiple animals within complex social and environmental settings

Abstract

AbstractAccurate detection and tracking of animals across diverse environments are crucial for behavioral studies in various disciplines, including neuroscience. Recently, machine learning and computer vision techniques have become integral to the neuroscientist’s toolkit, enabling high-throughput behavioral studies. Despite advancements in localizing individual animals in simple environments, the task remains challenging in complex conditions due to intra-class visual variability and environmental diversity. These limitations hinder studies in ethologically- relevant conditions, such as when animals are concealed within nests or in obscured environments. Moreover, current tools are laborious and time-consuming to employ, requiring extensive, setup-specific annotation and model training/validation procedures. To address these challenges, we introduce the ’Detect Any Mouse Model’ (DAMM), a pretrained object detector for localizing mice in complex environments, capable of robust performance with zero to minimal additional training on new experimental setups. Our approach involves collecting and annotating a diverse dataset that encompasses single and multi-housed mice in various lighting conditions, experimental setups, and occlusion levels. We utilize the Mask R-CNN architecture for instance segmentation and validate DAMM’s performance with no additional training data (zero-shot inference) and with few examples for fine-tuning (few-shot inference). DAMM excels in zero- shot inference, detecting mice, and even rats, in entirely unseen scenarios and further improves with minimal additional training. By integrating DAMM with the SORT algorithm, we demonstrate robust tracking, competitively performing with keypoint-estimation-based methods. Finally, to advance and simplify behavioral studies, we made DAMM accessible to the scientific community with a user-friendly Python API, shared model weights, and a Google Colab implementation.SignificancePresent deep learning tools for animal localization require extensive laborious annotation and time-consuming training for the creation of setup-specific models, slowing scientific progress. Additionally, the effectiveness of these tools in naturalistic settings is impeded by visual variability of objects and environmental diversity, hindering animal detection in complex environments. Our study presents the ’Detect Any Mouse Model’ (DAMM), a robustly validated object detector designed for localizing mice in complex environments. DAMM excels in generalization, robustly performing with zero to minimal additional training on previously unseen setups and multi-animal scenarios. Its integration with the SORT algorithm permits robust tracking, competitively performing with keypoint-estimation-based tools. These developments, along with our dissemination of DAMM, mark a significant step forward in streamlining ethologically-relevant animal behavioral studies.