Feature Distribution-based Knowledge Distillation for Deep Neural Networks

In recent years. various compression methods and compact models have been actively proposed to solve the significant computational costs accompanied by the achievement of high accuracy in deep learning. In particular, the knowledge distillation (KD) technique, which achieves a significant network compression effect by using the information of large teacher networks for the training of small student networks, is receiving a lot of attention because it has high scalability and reusability compared to the development of a compact model with a new structure. In this paper, we propose feature distribution-based knowledge distillation (FDKD) that effectively transfers semantic information using only the distribution information of feature maps obtained by a simple operation. Experimental results show that the proposed method improves the accuracy by up to 5.26% and 1.38%, respectively, compared to the baseline (i.e., Vanilla) and the existing KD scheme.