Affiliation:
1. Kyushu University
2. University of Alberta
3. Centre for Frontier AI Research (CFAR), A*STAR, and Institute of High Performance Computing (IHPC), A*STAR
4. Meta AI
5. University of Alberta and The University of Tokyo
Abstract
Over the past few years, deep neural networks (DNNs) have achieved tremendous success and have been continuously applied in many application domains. However, during the practical deployment in industrial tasks, DNNs are found to be erroneous-prone due to various reasons such as overfitting and lacking of robustness to real-world corruptions during practical usage. To address these challenges, many recent attempts have been made to repair DNNs for version updates under practical operational contexts by updating weights (i.e., network parameters) through retraining, fine-tuning, or direct weight fixing at a neural level. Nevertheless, existing solutions often neglect the effects of neural network architecture and weight relationships across neurons and layers. In this work, as the first attempt, we initiate to repair DNNs by jointly optimizing the architecture and weights at a higher (i.e., block level).
We first perform empirical studies to investigate the limitation of whole network-level and layer-level repairing, which motivates us to explore a novel repairing direction for DNN repair at the block level. To this end, we need to further consider techniques to address two key technical challenges, i.e.,
block localization
, where we should localize the targeted block that we need to fix; and how to perform
joint architecture and weight repairing
. Specifically, we first propose
adversarial-aware spectrum analysis for vulnerable block localization
that considers the neurons’ status and weights’ gradients in blocks during the forward and backward processes, which enables more accurate candidate block localization for repairing even under a few examples. Then, we further propose the
architecture-oriented search-based repairing
that relaxes the targeted block to a continuous repairing search space at higher deep feature levels. By jointly optimizing the architecture and weights in that space, we can identify a much better block architecture. We implement our proposed repairing techniques as a tool, named
ArchRepair
, and conduct extensive experiments to validate the proposed method. The results show that our method can not only repair but also enhance accuracy and robustness, outperforming the state-of-the-art DNN repair techniques.
Funder
JST-Mirai Program
JSPS KAKENHI
University Fellowships Toward the Creation of Science Technology Innovation
Canada CIFAR AI Chairs Program and the Natural Sciences and Engineering Research Council of Canada
Publisher
Association for Computing Machinery (ACM)