Exploring Static–Dynamic ID Matching and Temporal Static ID Inconsistency for Generalizable Deepfake Detection

Abstract

Identity‐based Deepfake detection methods have the potential to improve the generalization, robustness, and interpretability of the model. However, current identity‐based methods either require a reference or can only be used to detect face replacement but not face reenactment. In this paper, we propose a novel Deepfake video detection approach based on identity anomalies. We observe two types of identity anomalies: the inconsistency between clip‐level static ID (facial appearance) and clip‐level dynamic ID (facial behavior) and the temporal inconsistency of image‐level static IDs. Since these two types of anomalies can be detected through self‐consistency and do not depend on the manipulation type, our method is a reference‐free and manipulation‐independent approach. Specifically, our detection network consists of two branches: the static–dynamic ID discrepancy detection branch for the inconsistency between dynamic and static ID and the temporal static ID anomaly detection branch for the temporal anomaly of static ID. We combine the outputs of the two branches by weighted averaging to obtain the final detection result. We also designed two loss functions: the static–dynamic ID matching loss and the dynamic ID constraint loss, to enhance the representation and discriminability of dynamic ID. We conduct experiments on four benchmark datasets and compare our method with the state‐of‐the‐art methods. Results show that our method can detect not only face replacement but also face reenactment, and also has better detection performance over the state‐of‐the‐art methods on unknown datasets. It also has superior robustness against compression. Identity‐based features provide a good explanation of the detection results.