DiffQuant: Reducing Compression Difference for Neural Network Quantization

Abstract

Deep neural network quantization is a widely used method in the deployment of mobile or edge devices to effectively reduce memory overhead and speed up inference. However, quantization inevitably leads to a reduction in the performance and equivalence of models. Moreover, access to labeled datasets is often denied as they are considered valuable assets for companies or institutes. Consequently, performing quantization training becomes challenging without sufficient labeled datasets. To address these issues, we propose a novel quantization pipeline named DiffQuant, which can perform quantization training using unlabeled datasets. The pipeline includes two cores: the compression difference (CD) and model compression loss (MCL). The CD can measure the degree of equivalence loss between the full-precision and quantized models, and the MCL supports fine-tuning the quantized models using unlabeled data. In addition, we design a quantization training scheme that allows the quantization of both the batch normalization (BN) layer and the bias. Experimental results show that our method outperforms state-of-the-art methods on ResNet18/34/50 networks, maintaining performance with a reduced CD. We achieve Top-1 accuracies of 70.08%, 74.11%, and 76.16% on the ImageNet dataset for the 8-bit quantized ResNet18/34/50 models and reduce the gap to 0.55%, 0.61%, and 0.71% with the full-precision network, respectively. We achieve CD values of only 7.45%, 7.48%, and 8.52%, which allows DiffQuant to further exploit the potential of quantization.