Deep Convolutional Neural Network Compression based on the Intrinsic Dimension of the Training Data

Abstract

Selecting the optimal deep learning architecture for a particular task and dataset remains an ongoing challenge. Typically, this decision-making process involves exhaustive searches for neural network architectures or multi-phase optimization, which includes initial training, compression or pruning, and fine-tuning steps. In this study, we introduce an approach utilizing a deep reinforcement learning-based agent to dynamically compress a deep convolutional neural network throughout its training process. We integrate the concept of the intrinsic dimension of the training data to provide the agent with insights into the task's complexity. The agent employs two distinct ranking criteria, L1-norm-based and attention-based measures, to selectively prune filters from each layer as it determines necessary. In the experiments, we used the CIFAR-10 dataset and its subsets (2-class and 5-class subsets) to model the task complexity and showed that the agent learns different policies depending on the intrinsic dimension. The agent, on average, pruned off 78.48%, 77.9%, and 83.12% filters from all the layers of VGG-16 network for CIFAR-10 full, 5-class, and 2-class subsets respectively.