Hardware-Friendly Quantization via Outlier Scaling in Convolution-Attention-Based Hybrid Networks

Hybrid networks that combine convolution and attention have achieved state-of-the-art performance in computer vision tasks. Quantization is a promising compression method to efficiently utilize these hybrid networks in resource-constrained consumer electronics like IoT devices. However, although many hybrid networks have been proposed, research on quantization dedicated to hybrid networks has not yet been actively conducted. To bridge this gap, we propose a novel hardware-friendly post-training quantization method. Initially, we observe significant outliers in the bottleneck blocks of hybrid networks, which result in severe accuracy degradation due to quantization. To effectively address these outliers, we propose not only a novel outlier scaling method but also an objective function-based power-of-two approximation method that replaces conventional floating-point multiplication with hardware-friendly shift operations. To demonstrate the effectiveness of the proposed method, experiments were conducted on the ImageNet-1k dataset using a representative hybrid network, MobileViT. Our proposed method significantly mitigated the accuracy drop with a small parameter increase at the same model size compared to existing quantization methods.