Review of Recent Distillation Studies

Abstract

Knowledge distillation has gained a lot of interest in recent years because it allows for compressing a large deep neural network (teacher DNN) into a smaller DNN (student DNN), while maintaining its accuracy. Recent improvements have been made to knowledge distillation. One such improvement is the teaching assistant distillation method. This method involves introducing an intermediate "teaching assistant" model between the teacher and student. The teaching assistant is first trained to mimic the teacher, and then the student is trained to mimic the teaching assistant. This multi-step process can improve student performance. Another improvement to knowledge distillation is curriculum distillation. This method involves gradually training the student by exposing it to increasingly difficult concepts over time, similar to curriculum learning in humans. This process can help the student learn in a more stable and consistent manner. Finally, there is the mask distillation method. Here, the student is trained to specifically mimic the attention mechanisms learned by the teacher, not just the overall output of the teacher DNN. These improvements help to enhance the knowledge distillation process and enable the creation of more efficient DNNs.