Abstract
Visually-Rich Document Understanding (VRDU) has emerged as a critical field, driven by the need to automatically process documents containing complex visual, textual, and layout information. Recently, Multimodal Large Language Models (MLLMs) have shown remarkable potential in this domain, leveraging both Optical Character Recognition (OCR)-dependent and OCR-free frameworks to extract and interpret information in document images. This survey reviews recent advancements in MLLM-based VRDU, highlighting three core components: (1) methods for encoding and fusing textual, visual, and layout features; (2) training paradigms, including pretraining strategies, instruction-response tuning, and the trainability of different model modules; and (3) datasets utilized for pretraining, instruction-tuning, and supervised fine-tuning. Finally, we discuss the challenges and opportunities in this evolving field and propose future directions to advance the efficiency, generalizability, and robustness of VRDU systems.
Abstract (translated)
视觉丰富文档理解（VRDU）作为一项关键领域已经出现，这一领域的兴起源于自动处理包含复杂视觉、文本和布局信息的文档的需求。最近，多模态大型语言模型（MLLMs）在该领域展现了显著潜力，通过依赖光学字符识别（OCR）和不依赖OCR的框架来提取并解释文档图像中的信息。这篇综述回顾了基于MLLM的VRDU领域的近期进展，并重点介绍了三个核心组成部分：(1) 文本、视觉和布局特征编码及融合的方法；(2) 训练范式，包括预训练策略、指令-响应调优以及不同模型模块的可训练性；(3) 用于预训练、指令调优和监督微调的数据集。最后，我们讨论了这一不断发展的领域面临的挑战与机遇，并提出未来推进VRDU系统效率、泛化能力和鲁棒性的方向。
URL
https://arxiv.org/abs/2507.09861
