3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking
-
Published:2024-05-07
Issue:
Volume:
Page:
-
ISSN:0920-5691
-
Container-title:International Journal of Computer Vision
-
language:en
-
Short-container-title:Int J Comput Vis
Author:
Waldmann UrsORCID, Chan Alex Hoi HangORCID, Naik HemalORCID, Nagy MátéORCID, Couzin Iain D.ORCID, Deussen OliverORCID, Goldluecke BastianORCID, Kano FumihiroORCID
Abstract
AbstractMarkerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.
Funder
Deutsche Forschungsgemeinschaft Bundesministerium für Bildung und Forschung
Publisher
Springer Science and Business Media LLC
Reference93 articles.
1. Altmann, J. (1974). Observational study of behavior: Sampling methods. Behaviour, 49(3–4), 227–266. 2. An, L., Ren, J., Yu, T., Hai, T., Jia, Y., & Liu, Y. (2023). Three-dimensional surface motion capture of multiple freely moving pigs using mammal. Nature Communications, 14(1), 7727. 3. Anderson, D., & Perona, P. (2014). Toward a science of computational ethology. Neuron, 84(1), 18–31. 4. Badger, M. , Wang, Y. , Modh, A. , Perkes, A. , Kolotouros, N. , Pfrommer, B.G. , & Daniilidis, K. (2020). 3d bird reconstruction: A dataset, model, and shape recovery from a single view. In European conference on computer vision (pp. 1–17). 5. Bala, P. C., Eisenreich, B. R., Yoo, S. B. M., Hayden, B. Y., Park, H. S., & Zimmermann, J. (2020). Automated markerless pose estimation in freely moving macaques with openmonkeystudio. Nature Communication, 11, 4560.
Cited by
3 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
|
|