Evasion and Countermeasures Techniques to Detect Dynamic Binary Instrumentation Frameworks

Abstract

Dynamic Binary Instrumentation (DBI) is a dynamic analysis technique that allows arbitrary code to be executed when a program is running. DBI frameworks have started to be used to analyze malicious applications. As a result, different approaches have merged to detect and avoid them. Commonly referred to as split personality malware or evasive malware are pieces of malicious software that incorporate snippets of code to detect when they are under DBI framework analysis and thus mimic benign behavior. Recent studies have questioned the use of DBI in malware analysis, arguing that it increases the attack surface. In this article, we examine the anti-instrumentation techniques that abuse desktop-based DBI frameworks and existing countermeasures to determine if it is possible to reduce the exploitable attack surface introduced by these DBI frameworks. In particular, we review the related literature to identify (i) the existing set of DBI framework evasion techniques and (ii) the existing set of countermeasures to avoid them. We also analyze and compare the taxonomies introduced in the literature, and propose a new taxonomy that expands and completes the previous taxonomies. Our findings demonstrate that despite advances in DBI framework protections that make them quite suitable for system security purposes, more efforts are needed to reduce the attack surface that they add during application analysis. Only 12 of the 26 evasion techniques covered in this document have countermeasures, threatening the transparency of DBI frameworks. Furthermore, the impact in terms of performance overhead and effectiveness of these countermeasures in real-world situations is unknown. Finally, there are only proofs of concept for 9 of these 26 techniques, which makes it difficult to validate and study how they evade the analysis in order to counter them. We also point out some relevant issues in this context and outline ways of future research directions in the use of DBI frameworks for system security purposes.