Weight-adaptive joint mixed-precision quantization and pruning for neural network-based equalization in short-reach direct detection links

Abstract

Neural network (NN)-based equalizers have been widely applied for dealing with nonlinear impairments in intensity-modulated direct detection (IM/DD) systems due to their excellent performance. However, the computational complexity (CC) is a major concern that limits the real-time application of NN-based receivers. In this Letter, we propose, to our knowledge, a novel weight-adaptive joint mixed-precision quantization and pruning approach to reduce the CC of NN-based equalizers, where only integer arithmetic is taken into account instead of floating-point operations. The NN connections are either directly cutoff or represented by a proper number of quantization bits by weight partitioning, leading to a hybrid compressed sparse network that computes much faster and consumes less hardware resources. The proposed approach is verified in a 50-Gb/s 25-km pulse amplitude modulation (PAM)-4 IM/DD link using a directly modulated laser (DML) in the C-band. Compared with the traditional fully connected NN-based equalizer operated with standard floating-point arithmetic, about 80% memory can be saved at a minimum network size without degrading the system performance. Quantization is also shown to be more suitable to over-parameterized NN-based equalizers compared with NNs selected at a minimum size.