Metamorphic Testing of Image Classification and Consistency Analysis Using Clustering

Abstract

Testing deep learning systems requires expensive labeled data. In recent years, researchers began to leverage metamorphic testing to address this issue. However, metamorphic relations on image data remain poorly understood. To gain a deeper understanding of these metamorphic relations, we survey common image operations modeling covariate shift, manually classify and categorize the underlying metamorphic relations, and conduct experiments to validate our classifications. In our experiments, we train three popular convolutional neural network architectures on an image classification task. Next, we apply metamorphic operations on input test images and measure the change in classification accuracy and cross-entropy loss. A hierarchical clustering algorithm cluster these results and plots a dendrogram. We compare the groups from manual classification and the clusters from the algorithm to provide key insights. We find that Affine and Noise relations are consistent. Furthermore, we recommend metamorphic relationships to save time and better test deep learning systems in the future.