Abstract
Recent advances in large vision-language models (LVLMs) have enabled a new paradigm of end-to-end document image parsing, excelling in Optical Character Recognition (OCR) tasks such as text, table, and formula recognition. However, generative LVLMs, similarly to large language models (LLMs), are prone to hallucinations--generating words that do not exist in input images. Furthermore, LVLMs are designed for general purposes and tend to be less effective on OCR tasks compared to expert models that are trained on domain-specific datasets. In this paper, we propose DianJin-OCR-R1, a reasoning-enhanced framework designed to address these limitations through training reasoning-and-tool interleaved VLMs. Given a recognition instruction, our DianJin-OCR-R1 model first recognizes the content in the input image by its own OCR capabilities, and then calls other tools (i.e., other expert models) to obtain their results as references, finally looks again the image and rethinks about the reasoning process to provide the final recognized content. Since architectures of expert models are tailored for specific OCR tasks, which makes them less prone to hallucinations, their results can help VLMs mitigate hallucinations. Additionally, expert models are typically smaller in scale and easy to iterate, enabling performance improvements for VLMs at a lower cost. We evaluate our model on ReST and OmniDocBench, and experimental results show that our DianJin-OCR-R1 models consistently outperform their non-reasoning counterparts and expert OCR models, which proves the effectiveness of our method.
Abstract (translated)
近期在大型视觉语言模型（LVLM）方面的进展，已经开启了端到端文档图像解析的新范式，在光学字符识别（OCR）任务如文本、表格和公式识别方面表现出色。然而，生成式的LVLM与大型语言模型（LLM）一样容易产生幻觉——即生成那些不存在于输入图像中的词语。此外，LVLM由于是为通用目的设计的，所以在OCR任务上的表现往往不如专门针对特定领域数据集训练的专业模型。 在本文中，我们提出了DianJin-OCR-R1框架，这是一个增强推理能力的设计方案，旨在通过培训具备推理与工具交织功能的视觉语言模型来解决上述限制。给定识别指令后，我们的DianJin-OCR-R1模型首先利用自身的OCR能力识别输入图像中的内容，然后调用其他专业模型（即工具）获取参考结果，并最终再次审视图像以重新思考整个推理过程，从而提供最后的内容识别结果。 由于专业模型的架构是专门针对特定OCR任务定制设计的，因此它们产生幻觉的可能性较低。这些模型的结果可以帮助LVLM减少幻觉现象的发生。此外，专业模型通常规模较小且易于迭代，这使得在不增加成本的情况下提高LVLM性能成为可能。 我们在ReST和OmniDocBench数据集上对我们的模型进行了评估，实验结果表明，DianJin-OCR-R1模型在与非推理版本的比较中以及与专业OCR模型相比均表现出更优的效果，证明了我们方法的有效性。
URL
https://arxiv.org/abs/2508.13238
