Abstract
Recent advances in Visually-rich Document Understanding rely on large Vision-Language Models like Donut, which perform document-level Visual Question Answering without Optical Character Recognition. Despite their effectiveness, these models are too costly for real-time or resource-constrained applications. We investigate model compression through knowledge distillation, training compact student models from a larger teacher. We leverage mechanistic interpretability to drive student architecture design within this framework. By analyzing internal computations, we identify essential subcomponents to retain, while having a clear view of which subcomponents should be approximated, skipped, or reparametrized based on their function. This approach yields Donut-MINT (Mechanistic Interpretability-based Network Trimming), a pruned Donut variant that reduces inference time and memory usage while maintaining strong performance on DocVQA, a standard benchmark for document Visual Question Answering. Our method reframes compression as circuit discovery, bridging interpretability research and practical Vision-Language Model deployment.
Abstract (translated)
近期在丰富视觉文档理解方面取得的进展依赖于像Donut这样的大型视觉-语言模型，这些模型能够在没有光学字符识别的情况下执行基于文档级别的视觉问答任务。尽管这些模型非常有效，但它们对于实时应用或资源受限的应用来说成本过高。我们通过知识蒸馏进行模型压缩的研究，即从一个较大的教师模型训练出更加紧凑的学生模型，并利用机制性可解释性来指导学生网络架构的设计。通过分析内部计算过程，我们可以识别出需要保留的关键子组件，同时明确哪些子组件应该被近似、省略或重新参数化，这取决于它们的功能。 这种策略产生了Donut-MINT（基于机制性可解释性的网络修剪），这是一种经过优化的Donut变体，它在减少推理时间和内存使用的同时，在文档视觉问答的标准基准DocVQA上保持了强大的性能。我们的方法将压缩问题重新定义为电路发现，并且连接了可解释性研究和实用的视觉-语言模型部署之间的桥梁。
URL
https://arxiv.org/abs/2509.26235
