Building a Compact Convolutional Neural Network for Embedded Intelligent Sensor Systems Using Group Sparsity and Knowledge Distillation

Abstract

As artificial intelligence (AI)- or deep-learning-based technologies become more popular,the main research interest in the field is not only on their accuracy, but also their efficiency, e.g., theability to give immediate results on the users’ inputs. To achieve this, there have been many attemptsto embed deep learning technology on intelligent sensors. However, there are still many obstacles inembedding a deep network in sensors with limited resources. Most importantly, there is an apparenttrade-off between the complexity of a network and its processing time, and finding a structure witha better trade-off curve is vital for successful applications in intelligent sensors. In this paper, wepropose two strategies for designing a compact deep network that maintains the required level ofperformance even after minimizing the computations. The first strategy is to automatically determinethe number of parameters of a network by utilizing group sparsity and knowledge distillation (KD)in the training process. By doing so, KD can compensate for the possible losses in accuracy causedby enforcing sparsity. Nevertheless, a problem in applying the first strategy is the unclarity indetermining the balance between the accuracy improvement due to KD and the parameter reductionby sparse regularization. To handle this balancing problem, we propose a second strategy: a feedbackcontrol mechanism based on the proportional control theory. The feedback control logic determinesthe amount of emphasis to be put on network sparsity during training and is controlled based onthe comparative accuracy losses of the teacher and student models in the training. A surprising facthere is that this control scheme not only determines an appropriate trade-off point, but also improvesthe trade-off curve itself. The results of experiments on CIFAR-10, CIFAR-100, and ImageNet32 X 32datasets show that the proposed method is effective in building a compact network while preventingperformance degradation due to sparsity regularization much better than other baselines.