Enhancing Few-Shot Learning in Lightweight Models via Dual-Faceted Knowledge Distillation
Author:
Zhou Bojun1ORCID, Cheng Tianyu2, Zhao Jiahao1, Yan Chunkai1, Jiang Ling1ORCID, Zhang Xinsong2, Gu Juping13ORCID
Affiliation:
1. School of Information Science and Technology, Nantong University, Nantong 226019, China 2. School of Electrical Engineering, Nantong University, Nantong 226019, China 3. School of Electronic & Information Engineering, Suzhou University of Science and Technology, Suzhou 215009, China
Abstract
In recent computer vision research, the pursuit of improved classification performance often leads to the adoption of complex, large-scale models. However, the actual deployment of such extensive models poses significant challenges in environments constrained by limited computing power and storage capacity. Consequently, this study is dedicated to addressing these challenges by focusing on innovative methods that enhance the classification performance of lightweight models. We propose a novel method to compress the knowledge learned by a large model into a lightweight one so that the latter can also achieve good performance in few-shot classification tasks. Specifically, we propose a dual-faceted knowledge distillation strategy that combines output-based and intermediate feature-based methods. The output-based method concentrates on distilling knowledge related to base class labels, while the intermediate feature-based approach, augmented by feature error distribution calibration, tackles the potential non-Gaussian nature of feature deviations, thereby boosting the effectiveness of knowledge transfer. Experiments conducted on MiniImageNet, CIFAR-FS, and CUB datasets demonstrate the superior performance of our method over state-of-the-art lightweight models, particularly in five-way one-shot and five-way five-shot tasks.
Funder
National Natural Science Foundation of China Key Research & Development Program of Jiangsu Province
Reference50 articles.
1. Xie, J., Long, F., Lv, J., Wang, Q., and Li, P. (2022, January 19–24). Joint distribution matters: Deep Brownian distance covariance for few-shot classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA. 2. Bateni, P., Goyal, R., Masrani, V., Wood, F., and Sigal, L. (2020, January 13–19). Improved few-shot visual classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA. 3. A computer vision system for monitoring disconnect switches in distribution substations;Bogdan;IEEE Trans. Power Deliv.,2022 4. Liu, Y., Zhang, W., Xiang, C., Zheng, T., Cai, D., and He, X. (2022, January 19–24). Learning to affiliate: Mutual centralized learning for few-shot classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA. 5. Yang, Z., Wang, J., and Zhu, Y. (2022, January 23–27). Few-shot classification with contrastive learning. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.
|
|