Hardware-aware Network Compression for Hybrid Vision Transformer via Low-Rank Approximation

In recent years, transformer-based models have achieved excellent performance in various fields such as computer vision and language processing. Specifically, vision transformer (ViT) models outperform conventional convolutional neural networks (CNNs) in image classification tasks by achieving higher accuracy. However, ViT-based models often require more parameters than CNNs, making efficient deployment challenging in memory-constrained environments such as mobile devices. For example, the peak memory required in the output header layer of EfficientViT was 39.32 Mb. Deploying such a layer on the Zynq-7000 XC7Z045 FPGA board requires off-chip memory access, leading to inefficient power consumption. To address these issues, we applied a low-rank approximation method to reduce the memory requirements of the EfficientViT-B1 model. Our proposed method using the EfficientViT-B1 model on the ImageNet dataset achieved performance with only a 0.43% accuracy drop without requiring DRAM access.