PriME: PIM-Aware Efficient Compression for Memory-Bound Embedding Layers in sLLMs

With the growing demand for on-device AI, increasing efforts have been directed toward deploying lightweight small-scale large language models (sLLMs) on edge and mobile devices to enhance inference performance while minimizing computational cost and latency. As the number of decoder layers in sLLMs decreases, the embedding layer constitutes a substantial portion of the model's overall parameters and memory consumption. Consequently, efficient data compression is crucial; however, existing methods, such as quantization and pruning, drastically degrade accuracy when applied to the error-sensitive embedding layers. Moreover, embedding layer computations exhibit low arithmetic intensity (operations per byte), rendering them memory-bound. This limitation necessitates a shift from conventional von Neumann architectures to processing-in-memory (PIM) architectures. To address these challenges, this paper proposes (1) XOR-based Masking Compression (XMC), a lossless compression algorithm specialized for sLLM embedding layers, and (2) PriME, which integrates XMC with PIM architecture to alleviate memory bottlenecks in embedding layers. XMC enhances zero-bit representation in 16-bit FP data using ADD and XOR masking, achieving an average compression ratio of 1.49 × while being implementable with a 3-cycle decompression delay. PriME enables parallel processing of compressed data within PIM, accelerating embedding computations by an average of 4.0 × and up to 5.26 × compared to GPUs, while simultaneously reducing energy consumption by over 30 %, leading to an average energy efficiency improvement of 6.29 ×. Designed for broad applicability, PriME is compatible with various sLLMs and holds scalability for extension to multimodal small vision language models, demonstrating its versatility for efficient AI acceleration.