AE-Guide: Attention and Entropy Guided Visual Token Dropping for Accelerating High-Resolution Vision-Language Models

High-resolution vision-language models (VLMs) have emerged as powerful tools for multimodal tasks, but their reliance on numerous visual tokens from high-resolution inputs creates significant computational bottlenecks, especially in resource-constrained environments. Addressing this challenge, we introduce a novel, plug-and-play token-dropping method, engineered to operate within a predefined token budget. Our primary contribution is a refined visual token selection strategy that overcomes the limitations of traditional attention-guided methods. In contrast to traditional attention-guided approaches that leverage vision transformer (ViT) classification token attention for visual content measurement and budget allocation across image partitions, our approach incorporates the concept of token entropy to precisely identify the intrinsic information density and discriminative power of individual visual tokens. By capturing the inherent uncertainty within attention distributions, our method ensures that only the most salient and non-redundant visual tokens are preserved. We demonstrate that our proposed method effectively balances efficiency and accuracy across diverse VLM tasks. Empirical evaluations, particularly on the powerful LLaVA 1.6 Vicuna 13B model with a 20% token budget, show notable improvements. Compared to the attention-guided method, our approach achieved a 0.20% improvement in ScienceQA accuracy, a 9.57 point increase for MME, and a 0.09% gain in MMBench.