Abstract
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
Abstract (translated)
跨模态大型语言模型（Omni-LLMs）在音频视频理解任务中展示了强大的能力，但它们依赖于长的多模态令牌序列导致了显著的计算开销。尽管存在这一挑战，为Omni-LLM设计的令牌压缩方法仍然有限。为了弥合这一差距，我们提出了OmniSIFT（跨模态时空信息细粒度令牌压缩），这是一种针对Omni-LLMs定制的模态非对称令牌压缩框架。具体而言，OmniSIFT采用了一种两阶段压缩策略：(i) 一个时空视频修剪模块，移除来自帧内结构和帧间重叠的视频冗余；(ii) 一个视觉引导的音频选择模块，过滤音频令牌。整个框架通过可微分直通估计器进行端到端优化。 在五个代表性基准上的广泛实验展示了OmniSIFT的有效性和鲁棒性。值得注意的是，对于Qwen2.5-Omni-7B模型，虽然仅引入了4.85M参数，但OmniSIFT保持了比无训练基线（如OmniZip）更低的延迟。在仅有原始令牌上下文的25%的情况下，OmniSIFT始终超越所有压缩基线，并且在若干任务上甚至超过了完整令牌模型的表现。
URL
https://arxiv.org/abs/2602.04804
