Abstract
Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.
Abstract (translated)
多模态大型语言模型（MLLMs）在文档图像任务中表现出色，尤其是在光学字符识别（OCR）方面。然而，它们在处理文档图像机器翻译（DIMT）时遇到困难，因为这需要同时应对跨模式和跨语言的挑战。之前通过在DIMT数据集上进行监督微调(SFT)来增强DIMT能力的方法往往会遗忘模型原有的单语技能，如OCR。为了解决这些挑战，我们引入了一种新的微调范式——同步自审（SSR），旨在保持其OCR专业技能，灵感来自“双语认知优势”这一概念。具体来说，SSR提示模型在生成翻译文本之前先生成OCR文本，这使得模型能够利用其强大的单语言OCR能力同时学习跨语言的文本翻译。全面的实验表明，所提出的SSR学习有助于缓解灾难性遗忘，从而提高MLLMs在OCR和DIMT任务上的泛化能力。
URL
https://arxiv.org/abs/2507.08309
