A Malware and Variant Detection Method Using Function Call Graph Isomorphism

Abstract

The huge influx of malware variants are generated using packing and obfuscating techniques. Current antivirus software use byte signature to identify known malware, and this method is easy to be deceived and generally ineffective for identifying malware variants. Antivirus experts use hash signature to verify if captured sample is one of the malware databases, and this method cannot recognize malware variants whose hash signatures have changed completely. Function call graph is a high-level abstraction representation of a program and more stable and resilient than byte or hash signature. In this paper, function call graph is used as signature of a program, and two kinds of graph isomorphism algorithms are employed to identify known malware and its variants. Four experiments are designed to evaluate the performance of the proposed method. Experimental results indicate that the proposed method is effective and efficient for identifying known malware and a portion of their variants. The proposed method can also be used to index and locate a large-scale malware database and group malware to the corresponding family.