Efficient MXINT4 Inference via Hardware-Based Dynamic Sparsity Detection and Skip-Activation

Microscaling (MX) quantization has recently garnered significant attention due to its ability to achieve high compression ratios while preserving critical information across various deep learning tasks. It has been officially adopted and deployed across diverse hardware platforms. However, the blockwise processing required during MX quantization introduces considerable overhead, resulting in substantial latency when executed on conventional hardware. Consequently, it becomes challenging to realize actual computational acceleration using standard architectures. This highlights the necessity for a dedicated hardware design tailored to MX quantization, capable of mitigating such overhead and enabling practical speedups. In this paper, we present a hardware quantizer module specifically designed for the MX format and propose the MX quantizer with dynamic sparsity detection (MQDS), which identifies activation sparsity during the quantization process. Furthermore, we introduce a sparse computation flow that leverages the outputs of MQDS to address the area overhead and latency compatibility issues inherent in conventional MX quantization approaches. Experimental results demonstrate that MQDS achieves an average activation skip detection ratio of 66.49 % on vision transformer models while occupying only 12.2 % of the area compared to traditional integer quantization designs.