RoPE on the Fast Track: Latency-Optimized LLM Inference on GPUs with Low-Rank QKV Factorization and Rotary Positional Embedding

In large language model (LLM) inference, rotary positional embedding (RoPE) contributes significantly to overall decoding latency due to accumulated overhead from numerous micro-kernel invocations and its sequential execution after Q and K generation, which prevents effective parallelization. Additionally, the Q,K, and V (QKV) generation weights in modern LLMs-where the model dimension reaches several thou-sand-are each several gigabytes in size, making it impractical to store them in the on-chip memory of FPGAs or custom ASICs. This necessitates repeated off-chip accesses, limiting hardware utilization and inference speed. To address these bottlenecks, we apply rank =8 factorization to the QKV generation weights, reducing memory usage by over 99% and lowering per-token FLOPs to 1.41% of the baseline, with minimal loss in accuracy. Furthermore, we reformulate RoPE as a Weight-side RoPE mechanism by pre-applying it to low-rank weights and enabling its parallel execution with RMS normalization on the GPU. This structural change improves execution efficiency by enabling concurrent computation. Experimental results using a Llama3based model on an RTX 3090 GPU show that decoding latency is reduced by 1.14 × for batch size 1 and 1.17 × for batch size 2, while maintaining comparable accuracy to the baseline.