Affiliation:
1. Peking University, China
2. University of Alberta, Canada
3. Nanyang Technological University, Singapore
Abstract
Deep learning (DL) has recently been widely applied to diverse source code processing tasks in the software engineering (SE) community, which achieves competitive performance (e.g., accuracy). However, the robustness, which requires the model to produce consistent decisions given minorly perturbed code inputs, still lacks systematic investigation as an important quality indicator. This article initiates an early step and proposes a framework CARROT for robustness detection, measurement, and enhancement of DL models for source code processing. We first propose an optimization-based attack technique CARROT
A
to generate valid adversarial source code examples effectively and efficiently. Based on this, we define the robustness metrics and propose robustness measurement toolkit CARROT
M
, which employs the worst-case performance approximation under the allowable perturbations. We further propose to improve the robustness of the DL models by adversarial training (CARROT
T
) with our proposed attack techniques. Our in-depth evaluations on three source code processing tasks (i.e., functionality classification, code clone detection, defect prediction) containing more than 3 million lines of code and the classic or SOTA DL models, including GRU, LSTM, ASTNN, LSCNN, TBCNN, CodeBERT, and CDLH, demonstrate the usefulness of our techniques for ❶ effective and efficient adversarial example detection, ❷ tight robustness estimation, and ❸ effective robustness enhancement.
Funder
National Key R&D Program of China
National Natural Science Foundation of China
Canada CIFAR AI Program
NSERC Discovery Grant of Natural Sciences and Engineering Research Council of Canada
JSPS KAKENHI
JST-Mirai Program
Publisher
Association for Computing Machinery (ACM)
Reference99 articles.
1. A Survey of Machine Learning for Big Code and Naturalness
2. Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. 2016. A convolutional attention network for extreme summarization of source code. In Proceedings of the 33rd International Conference on Machine Learning. JMLR.org, 2091–2100. Retrieved from http://proceedings.mlr.press/v48/allamanis16.html.
3. Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating sequences from structured representations of code. In Proceedings of the 7th International Conference on Learning Representations. OpenReview.net. Retrieved from https://openreview.net/forum?id=H1gKYo09tX.
4. code2vec: learning distributed representations of code
5. Generating Natural Language Adversarial Examples
Cited by
11 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Detecting Overfitting of Machine Learning Techniques for Automatic Vulnerability Detection;Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering;2023-11-30
2. An Extensive Study on Adversarial Attack against Pre-trained Models of Code;Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering;2023-11-30
3. Broken Promises: Measuring Confounding Effects in Learning-based Vulnerability Discovery;Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security;2023-11-26
4. How Important are Good Method Names in Neural Code Generation? A Model Robustness Perspective;ACM Transactions on Software Engineering and Methodology;2023-10-23
5. On-the-fly Improving Performance of Deep Code Models via Input Denoising;2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE);2023-09-11