MoRAM: A Hybrid Framework for Gradient-based Low-rank Adaptation

Recent advances in foundation models have enabled the transfer of prior knowledge to a wide range of downstream tasks. However, fully fine-tuning these models remains impractical in many scenarios due to substantial memory and storage requirements. These challenges are especially pronounced in resource-constrained physical systems. Parameter-efficient fine-tuning techniques address this issue by updating only a small subset of parameters, enabling efficient adaptation while preserving the pretrained model’s original knowledge. A prominent class of parameter-efficient fine-tuning techniques includes parameter selection and reparameterization. Nevertheless, existing parameter selection methods often suffer from inefficient memory usage, primarily due to the need to store dense binary masks that identify selected parameters. To address this limitation, we propose masked low-rank matrix multiplication (MoRAM), a novel framework that combines gradient-based parameter selection with low-rank matrix decomposition. By maintaining only sparse parameter indices and low-rank matrices, MoRAM achieves a significant memory usage reduction compared to prior methods while retaining comparable task performance on several benchmark datasets. These results demonstrate MoRAM’s effectiveness and scalability, making it a promising approach for efficient model adaptation in constrained environments.