RNNCon: Contribution Coverage Testing for Stacked Recurrent Neural Networks
Author:
Du Xiaoli12ORCID, Zeng Hongwei12, Chen Shengbo12, Lei Zhou12
Affiliation:
1. School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China 2. Shanghai Key Laboratory of Computer Software Testing and Evaluating, Shanghai 201112, China
Abstract
Recurrent Neural Networks (RNNs) are applied in safety-critical fields such as autonomous driving, aircraft collision detection, and smart credit. They are highly susceptible to input perturbations, but little research on RNN-oriented testing techniques has been conducted, leaving a threat to a large number of sequential application domains. To address these gaps, improve the test adequacy of RNNs, find more defects, and improve the performance of RNNs models and their robustness to input perturbations. We aim to propose a test coverage metric for the underlying structure of RNNs, which is used to guide the generation of test inputs to test RNNs. Although coverage metrics have been proposed for RNNs, such as the hidden state coverage in RNN-Test, they ignore the fact that the underlying structure of RNNs is still a fully connected neural network but with an additional “delayer” that records the network state at the time of data input. We use the contributions, i.e., the combination of the outputs of neurons and the weights they emit, as the minimum computational unit of RNNs to explore the finer-grained logical structure inside the recurrent cells. Compared to existing coverage metrics, our research covers the decision mechanism of RNNs in more detail and is more likely to generate more adversarial samples and discover more flaws in the model. In this paper, we redefine the contribution coverage metric applicable to Stacked LSTMs and Stacked GRUs by considering the joint effect of neurons and weights in the underlying structure of the neural network. We propose a new coverage metric, RNNCon, which can be used to guide the generation of adversarial test inputs. And we design and implement a test framework prototype RNNCon-Test. 2 datasets, 4 LSTM models, and 4 GRU models are used to verify the effectiveness of RNNCon-Test. Compared to the current state-of-the-art study RNN-Test, RNNCon can cover a deeper decision logic of RNNs. RNNCon-Test is not only effective in identifying defects in Deep Learning (DL) systems but also in improving the performance of the model if the adversarial inputs generated by RNNCon-Test are filtered and added to the training set to retrain the model. In the case where the accuracy of the model is already high, RNNCon-Test is still able to improve the accuracy of the model by up to 0.45%.
Funder
State Key Laboratory of Computer Architecture Institute of Computing Technology, Chinese Academy of Sciences and the High Performance Computing Center, Shanghai University Shanghai Engineering Research Center of Intelligent Computing System
Subject
General Physics and Astronomy
Reference47 articles.
1. Collobert, R., and Weston, J. (2008, January 5–9). A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland. 2. Mikolov, T., Chen, K., Corrado, G.S., and Dean, J. (2013, January 2–4). Efficient Estimation of Word Representations in Vector Space. Proceedings of the International Conference on Learning Representations, Scottsdale, AZ, USA. 3. ImageNet Classification with Deep Convolutional Neural Networks;Krizhevsky;Commun. ACM,2017 4. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups;Hinton;IEEE Signal Process. Mag.,2012 5. Chen, B., Jiang, J., Wang, X., Wan, P., Wang, J., and Long, M. (2022). Debiased Self-Training for Semi-Supervised Learning. arXiv.
|
|