Audio-Text Retrieval Based on Contrastive Learning and Collaborative Attention Mechanism

Abstract

Abstract Existing research on audio-text retrieval is limited by the size of the dataset and the structure of the network, making it difficult to learn the ideal featuresof audio and text resulting in low retrieval accuracy. In this paper, we construct an audio-text retrieval model based on contrastive learning and collaborative attention mechanism . We first reduce model overfitting by implementing audio augmentation strategies including adding Gaussian noise, adjusting the pitch and changing the time shift.Additionally, we design a co-attentive mechanism module that the audio data and text data guide each other in feature learning, effectively capturing the connection between the audio modality and the text modality. Finally we apply the contrastive learning methods between the augmented audio data and the original audio, allowing the model to effectively learn a richer set of audio features. The retrieval accuracy of our proposed model is significantly improved on publicly available datasets AudioCaps and Clotho.