Affiliation:
1. University of Connecticut, USA
Abstract
Network pruning is a widely used technique to reduce computation cost and model size for deep neural networks. However, the typical three-stage pipeline (i.e., training, pruning, and retraining (fine-tuning)) significantly increases the overall training time. In this article, we develop a systematic weight-pruning optimization approach based on surrogate Lagrangian relaxation (SLR), which is tailored to overcome difficulties caused by the discrete nature of the weight-pruning problem. We further prove that our method ensures fast convergence of the model compression problem, and the convergence of the SLR is accelerated by using quadratic penalties. Model parameters obtained by SLR during the training phase are much closer to their optimal values as compared to those obtained by other state-of-the-art methods. We evaluate our method on image classification tasks using CIFAR-10 and ImageNet with state-of-the-art multi-layer perceptron based networks such as MLP-Mixer; attention-based networks such as Swin Transformer; and convolutional neural network based models such as VGG-16, ResNet-18, ResNet-50, ResNet-110, and MobileNetV2. We also evaluate object detection and segmentation tasks on COCO, the KITTI benchmark, and the TuSimple lane detection dataset using a variety of models. Experimental results demonstrate that our SLR-based weight-pruning optimization approach achieves a higher compression rate than state-of-the-art methods under the same accuracy requirement and also can achieve higher accuracy under the same compression rate requirement. Under classification tasks, our SLR approach converges to the desired accuracy × faster on both of the datasets. Under object detection and segmentation tasks, SLR also converges 2× faster to the desired accuracy. Further, our SLR achieves high model accuracy even at the hardpruning stage without retraining, which reduces the traditional three-stage pruning into a two-stage process. Given a limited budget of retraining epochs, our approach quickly recovers the model’s accuracy.
Funder
Semiconductor Research Corporation (SRC) Artificial Intelligence Hardware program, the National Science Foundation
USDA-NIFA Agriculture and Food Research Initiative
Publisher
Association for Computing Machinery (ACM)
Subject
Electrical and Electronic Engineering,Computer Graphics and Computer-Aided Design,Computer Science Applications
Reference52 articles.
1. Yue Bai, Huan Wang, Zhiqiang Tao, Kunpeng Li, and Yun Fu. 2022. Dual lottery ticket hypothesis. In Proceedings of the International Conference on Learning Representations.
2. Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT ’10). 177–186.
3. Distributed optimization and statistical learning via the alternating direction method of multipliers;Boyd Stephen;Foundations and Trends in Machine Learning,2011
4. Survey on Lagrangian relaxation for MILP: Importance, challenges, historical review, recent advancements, and opportunities;Bragin Mikhail A.;Annals of Operations Research.,2023
5. Mikhail A. Bragin Peter B. Luh Bing Yan Tongxin Zheng Dane A. Schiro Feng Zhao and Jinye Zhao. 2022. Novel quality measure and efficient resolution of convex hull pricing for unit commitment. arXiv 2304.07990 (2022).