S$$^{2}$$P$$^{3}$$: Self-Supervised Polarimetric Pose Prediction
-
Published:2024-01-12
Issue:6
Volume:132
Page:2177-2194
-
ISSN:0920-5691
-
Container-title:International Journal of Computer Vision
-
language:en
-
Short-container-title:Int J Comput Vis
Author:
Ruhkamp PatrickORCID, Gao Daoyi, Navab Nassir, Busam Benjamin
Abstract
AbstractThis paper proposes the first self-supervised 6D object pose prediction from multimodal RGB + polarimetric images. The novel training paradigm comprises (1) a physical model to extract geometric information of polarized light, (2) a teacher–student knowledge distillation scheme and (3) a self-supervised loss formulation through differentiable rendering and an invertible physical constraint. Both networks leverage the physical properties of polarized light to learn robust geometric representations by encoding shape priors and polarization characteristics derived from our physical model. Geometric pseudo-labels from the teacher support the student network without the need for annotated real data. Dense appearance and geometric information of objects are obtained through a differentiable renderer with the predicted pose for self-supervised direct coupling. The student network additionally features our proposed invertible formulation of the physical shape priors that enables end-to-end self-supervised training through physical constraints of derived polarization characteristics compared against polarimetric input images. We specifically focus on photometrically challenging objects with texture-less or reflective surfaces and transparent materials for which the most prominent performance gain is reported.
Funder
Technische Universität München
Publisher
Springer Science and Business Media LLC
Reference52 articles.
1. Atkinson, G. A. (2017). Polarisation photometric stereo. Computer Vision and Image Understanding, 160, 158–167. 2. Atkinson, G. A., & Hancock, E. R. (2005). Multi-view surface reconstruction using polarization. In Tenth IEEE international conference on computer vision (ICCV’05) (Vol. 1, pp. 309–316). 3. Atkinson, G. A., & Hancock, E. R. (2006). Recovery of surface orientation from diffuse polarization. IEEE Transactions on Image Processing, 15(6), 1653–1664. 4. Ba, Y., Gilbert, A., Wang, F., Yang, J., Chen, R., Wang, Y., Yan, L., Shi, B., & Kadambi, A. (2020). Deep shape from polarization. In Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, proceedings, Part XXIV 16 (pp. 554–571). 5. Busam, B., Ruhkamp, P., Virga, S., Lentes, B., Rackerseder, J., Navab, N., Hennersperger, C. (2018). Markerless inside-out tracking for 3D ultrasound compounding. In Simulation, image processing, and ultrasound systems for assisted diagnosis and navigation (pp. 56–64). Springer.
|
|