Towards accurate knowledge transfer via target-awareness representation disentanglement

Abstract

AbstractFine-tuning deep neural networks pre-trained on large scale datasets is one of the most practical transfer learning paradigm given limited quantity of training samples. To obtain better generalization, using the starting point as the reference (SPAR), either through weights or features, has been successfully applied to transfer learning as a regularizer. However, due to the domain discrepancy between the source and target task, there exists obvious risk of negative transfer in a straightforward manner of knowledge preserving. In this paper, we propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement ($$\textrm{TRED}$$ TRED ), where the relevant knowledge with respect to the target task is disentangled from the original source model and used as a regularizer during fine-tuning the target model. Two alternative approaches, maximizing Maximum Mean Discrepancy (Max-MMD) and minimizing mutual information (Min-MI) are introduced to achieve the desired disentanglement. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average. $$\textrm{TRED}$$ TRED also outperforms related state-of-the-art transfer learning regularizers such as $$\mathrm {L^2\text {-}SP}$$ L 2 - SP , $$\textrm{AT}$$ AT , $$\textrm{DELTA}$$ DELTA , and $$\textrm{BSS}$$ BSS . Moreover, our solution is compatible with different choices of disentangling strategies. While the combination of Max-MMD and Min-MI typically achieves higher accuracy, only using Max-MMD can be a preferred choice in applications with low resource budgets.