CLS-Net: An Action Recognition Algorithm Based on Channel-Temporal Information Modeling

Abstract

The modeling of channel and temporal information is of crucial importance for action recognition tasks. To build a high-performance action recognition network by effectively capturing channel and temporal information, we propose CLS-Net: an action recognition algorithm based on channel-temporal information modeling. The proposed CLS-Net characterizes channel and temporal information by inserting multiple modules to an end-to-end backbone network, including a channel attention module (CA module) for modeling channel information, a long-term temporal module (LT module) and a short-term temporal module (ST module) for modeling temporal information. Specifically, the CA module extracts the correlation between feature channels so the network can learn to selectively strengthen the features containing useful information and suppress the useless features through global information. The LT module moves some channels in the temporal dimension to realize information interaction across time domains and model global temporal information. The ST module enhances the motion-sensitive features by calculating the feature-level frame difference information and realizes the representation of local motion information. Since the multi-module insertion mode directly affects the whole model’s final performance, we propose a novel multi-module insertion mode instead of a simple series or parallel connection to ensure that the multiple modules can complement one another and cooperate with each other more efficiently. CLS-Net achieves SOTA performance on the EgoGesture and Jester dataset in the same type of network and achieves competitive results on the Something-Something V2 dataset.