EvadeRL: Evading PDF Malware Classifiers with Deep Reinforcement Learning

Abstract

With the growing popularity of information digitization and the advancement of executable file detection technology, PDF has emerged as an important carrier of malicious documents. Despite the improved efficacy of machine learning-based classifiers in detecting PDF malware, adversaries have proposed a variety of countermeasures to evade detection, such as generating adversarial examples. In contrast to other peer works attempting to expose the vulnerability of learning-based detection models, this work addresses the deficiencies of existing research by pointing out that the stochastic manipulations they applied may be highly computationally demanding. This work proposed EvadeRL, a general framework for automatically generating adversarial examples based on double deep Q-Network. The details of EvadeRL are briefly described as follows. First, the EvadeRL agent chooses a series of actions to modify the given PDF files and uses the classification results, as well as observations returning from the environment, to calculate the approximate value of each action. Second, through the interaction of the agent and the environment, the experiences gained are stored to train the decision network. Finally, the agent can generate adversarial examples against the target detector by taking the optimal behaviors after training. This study also contributes to the sustainability of evasion attacks by online fine-tuning; to the best of our current knowledge, this is the first study in the field that focuses on evolving malware. The experiments reveal that EvadeRL obtains a high evasion rate against PDF malware detectors and outperforms other approaches in terms of execution cost, robustness against hardened detectors, and sustainability against evolving malware and detectors.