Merge-and-Split Graph Convolutional Network for Skeleton-Based Interaction Recognition

Abstract

We introduce an innovative approach to address a significant challenge in interaction recognition, specifically the capture of correlation features between different interaction body parts. These features are often overlooked by traditional graph convolution networks commonly used in interaction recognition tasks. Our solution, the Merge-and-Split Graph Convolutional Network, takes a unique perspective, treating interaction recognition as a global problem. It leverages a Merge-and-Split Graph structure to effectively capture dependencies between interaction body parts. To extract the essential interaction features, we introduce the Merge-and-Split Graph Convolution module, which seamlessly combines the Merge-and-Split Graph with Graph Convolutional Networks. This fusion enables the extraction of rich semantic information between adjacent joint points. In addition, we introduce a Short-term Dependence module designed to extract joint and motion characteristics specific to each type of interaction. Furthermore, to extract correlation features between different hierarchical sets, we present the Hierarchical Guided Attention Module. This module plays a crucial role in highlighting the relevant hierarchical sets that contain essential interaction information. The effectiveness of our proposed model is demonstrated by achieving state-of-the-art performance on 2 widely recognized datasets, namely, the NTU60 and NTU120 interaction datasets. Our model’s efficacy is rigorously validated through extensive experiments, and we have made the code available for the research community at https://github.com/wanghq05/MS-GCN/ .