A hybrid detection algorithm for 5G OTFS waveform for 64 and 256 QAM with Rayleigh and Rician channels

Abstract

Abstract Signal detection in orthogonal time frequency space modulation is one of the critical aspects for enhancing the throughput of the beyond fifth-generation framework, especially in challenging environments characterized by high mobility and multipath propagation. In this work, we proposed a hybrid detection algorithm by combining zero-forcing equalization (ZFE) and minimum square error equalization (MMSE), also known as ZFE–MMSE, with Rician and Rayleigh channels for 64 and 256 quadrature amplitude modulation. The combination of ZFE and MMSE allows for a more comprehensive approach to equalization. ZFE addresses channel-induced distortions, while MMSE tackles the impact of noise. Together, they enhance the overall performance of the equalization process, resulting in improved signal detection accuracy. The analysis and comparison of the simulation parameters with traditional detection systems include bit error rate, power spectral density (PSD), and complexity. The projected algorithms achieve a signal-to-noise ratio and a PSD gain of 2 dB and −800, respectively, outperforming the traditional detection techniques.