RCAT: Retentive CLIP Adapter Tuning for Improved Video Recognition

Abstract

The advent of Contrastive Language-Image Pre-training (CLIP) models has revolutionized the integration of textual and visual representations, significantly enhancing the interpretation of static images. However, their application to video recognition poses unique challenges due to the inherent dynamism and multimodal nature of video content, which includes temporal changes and spatial details beyond the capabilities of traditional CLIP models. These challenges necessitate an advanced approach capable of comprehending the complex interplay between the spatial and temporal dimensions of video data. To this end, this study introduces an innovative approach, Retentive CLIP Adapter Tuning (RCAT), which synergizes the foundational strengths of CLIP with the dynamic processing prowess of a Retentive Network (RetNet). Specifically designed to refine CLIP’s applicability to video recognition, RCAT facilitates a nuanced understanding of video sequences by leveraging temporal analysis. At the core of RCAT is its specialized adapter tuning mechanism, which modifies the CLIP model to better align with the temporal intricacies and spatial details of video content, thereby enhancing the model’s predictive accuracy and interpretive depth. Our comprehensive evaluations on benchmark datasets, including UCF101, HMDB51, and MSR-VTT, underscore the effectiveness of RCAT. Our proposed approach achieves notable accuracy improvements of 1.4% on UCF101, 2.6% on HMDB51, and 1.1% on MSR-VTT compared to existing models, illustrating its superior performance and adaptability in the context of video recognition tasks.