Multi-Step Training Framework Using Sparsity Training for Efficient Utilization of Accumulated New Data in Convolutional Neural Networks

With the accumulation of large amounts of data that have the most significant impact on network performance and the development of hardware accelerators that can support such data processing, convolutional neural networks (CNNs) have achieved remarkable advances in various applications such as computer vision. In particular, in recent years, various new data suitable for user environments have been obtained from numerous mobile devices, and accordingly, methods have to be developed to train CNNs with only new data on the existing pre-trained network without access to the data used for the pre-training process on server platforms. Nevertheless, only a few studies have considered efficient training schemes for new data. In this study, we propose a multi-step training framework that efficiently utilizes accumulated new data for CNN training. In detail, to significantly improve the performance of CNNs, the proposed method creates unimportant filters while preserving knowledge of previous data through selective sparsity training and can effectively utilize these filters for CNN training of new data through diverse re-initialization and adaptive online distillation techniques. In addition, the proposed multi-step sparsity training with multi-step loss enables iterative network training of new data. The results of extensive experiments performed on various datasets show that the proposed method outperforms many existing methods, including fine-tuning.