PoseR - A deep learning toolbox for decoding animal behavior

Abstract

AbstractThe actions of animals provide a window into how their minds work. Recent advances in deep learning are providing powerful approaches to recognize patterns of animal movement from video recordings, including markerless pose estimation models. However, tools to efficiently parse coordinates of animal position and pose into meaningful semantic behavioral labels are lacking. Here, we present PoseRecognition (PoseR), a behavioral decoder leveraging state- of-the-art action recognition models using spatio-temporal graph convolutional networks. We show that it can be used to decode animal behavior quickly and accurately from pose estimations, using zebrafish larvae and mice as model organisms. PoseR can be accessed using a Napari plugin, which facilitates efficient behavioral extraction, annotation, model training and deployment. We have simplified the workflow of behavioral analysis after pose estimation, transforming coordinates of animal position and pose into meaningful semantic behavioral labels, using methods designed for fast and accurate behavioral extraction, annotation, model training and deployment. Furthermore, we contribute a novel method for unsupervised clustering of behaviors and provide open-source access to our zebrafish datasets and models. The design of our tool ensures scalability and versatility for use across multiple species and contexts, improving the efficiency of behavioral analysis across fields.