Abstract
We introduce DEEVISum (Distilled Early Exit Vision language model for Summarization), a lightweight, efficient, and scalable vision language model designed for segment wise video summarization. Leveraging multi modal prompts that combine textual and audio derived signals, DEEVISum incorporates Multi Stage Knowledge Distillation (MSKD) and Early Exit (EE) to strike a balance between performance and efficiency. MSKD offers a 1.33% absolute F1 improvement over baseline distillation (0.5%), while EE reduces inference time by approximately 21% with a 1.3 point drop in F1. Evaluated on the TVSum dataset, our best model PaLI Gemma2 3B + MSKD achieves an F1 score of 61.1, competing the performance of significantly larger models, all while maintaining a lower computational footprint. We publicly release our code and processed dataset to support further research.
Abstract (translated)
我们介绍了DEEVISum（用于摘要的早期退出视觉语言模型），这是一种轻量级、高效且可扩展的视觉语言模型，专门设计用于分段视频摘要。通过结合文本和音频信号的多模态提示，DEEVISum 集成了多阶段知识蒸馏（MSKD）和早期退出（EE），以在性能与效率之间取得平衡。MSKD 相比基线蒸馏方法，在 F1 分数上提高了 1.33% 的绝对值（0.5%），而 EE 将推理时间减少了大约 21%，F1 分数下降了 1.3 点。在 TVSum 数据集上的评估中，我们的最佳模型 PaLI Gemma2 3B + MSKD 达到了 F1 得分 61.1，在保持较低计算负担的同时，与明显更大的模型性能相当。我们公开发布了代码和处理后的数据集以支持进一步的研究。
URL
https://arxiv.org/abs/2504.21831
