Rare Category Analysis for Complex Data: A Review

Abstract

Though the sheer volume of data that is collected is immense, it is the rare categories that are often the most important in many high-impact domains, ranging from financial fraud detection in online transaction networks to emerging trend detection in social networks, from spam image detection on social media platforms to rare disease diagnosis in medical decision support systems. The unique challenges of rare category analysis include (1) the highly skewed class distribution; (2) the non-separable nature of the rare categories from the majority classes; (3) data and task heterogeneity; and (4) the time-evolving property of the input data sources. This survey reviews state-of-the-art techniques used in complex rare category analysis, where the majority classes have a smooth distribution while the minority classes exhibit the compactness property in the feature space or subspace. Rare category analysis aims to identify, characterize, represent, and interpret anomalies that not only show statistical significance but also exhibit interesting patterns (e.g., compactness, high-order structures, showing in a burst). We introduce our study, define the problem setting, and describe the unique challenges of complex rare category analysis. We then present a comprehensive review of recent advances that are designed for this problem setting, from rare category exploration without any label information to rare category exposition that characterizes rare examples with a compact representation, from the representation of rare patterns in a salient embedding space to the interpretation the prediction results and providing relevant clues for the end-users’ interpretation. Finally we discuss potential challenges and shed light on the future directions for complex rare category analysis. 1