Abstract
Text Image Machine Translation (TIMT)-the task of translating textual content embedded in images-is critical for applications in accessibility, cross-lingual information access, and real-world document understanding. However, TIMT remains a complex challenge due to the need for accurate optical character recognition (OCR), robust visual-text reasoning, and high-quality translation, often requiring cascading multi-stage pipelines. Recent advances in large-scale Reinforcement Learning (RL) have improved reasoning in Large Language Models (LLMs) and Multimodal LLMs (MLLMs), but their application to end-to-end TIMT is still underexplored. To bridge this gap, we introduce MT$^{3}$, the first framework to apply Multi-Task RL to MLLMs for end-to-end TIMT. MT$^{3}$ adopts a multi-task optimization paradigm targeting three key sub-skills: text recognition, context-aware reasoning, and translation. It is trained using a novel multi-mixed reward mechanism that adapts rule-based RL strategies to TIMT's intricacies, offering fine-grained, non-binary feedback across tasks. Furthermore, to facilitate the evaluation of TIMT in authentic cross-cultural and real-world social media contexts, we introduced XHSPost, the first social media TIMT benchmark. Our MT$^{3}$-7B-Zero achieves state-of-the-art results on the latest in-domain MIT-10M benchmark, outperforming strong baselines such as Qwen2.5-VL-72B and InternVL2.5-78B by notable margins across multiple metrics. Additionally, the model shows strong generalization to out-of-distribution language pairs and datasets. In-depth analyses reveal how multi-task synergy, reinforcement learning initialization, curriculum design, and reward formulation contribute to advancing MLLM-driven TIMT.
Abstract (translated)
文本图像机器翻译（TIMT）——即对嵌入在图片中的文字内容进行翻译的任务，在可访问性、跨语言信息检索和现实世界文档理解的应用中至关重要。然而，由于需要准确的光学字符识别（OCR）、稳健的文字-视觉推理以及高质量的翻译，这一任务仍然面临诸多挑战，通常需采用多阶段级联管道来解决这些问题。近期的大规模强化学习（RL）在大型语言模型（LLMs）和多模态LLM（MLLMs）中的应用显著提升了这些模型的推理能力，但它们在端到端TIMT上的应用仍处于探索阶段。 为填补这一空白，我们引入了MT$^{3}$，这是一个首次将多任务强化学习应用于MLLM以实现端到端TIMT的框架。MT$^{3}$采用了一种面向三大关键子技能（文字识别、基于上下文的理解和翻译）的多任务优化范式，并通过一种新颖的混合奖励机制进行训练，该机制适应了TIMT特有的复杂性，并提供了跨任务的精细且非二元化的反馈。此外，为了评估在真正的跨国文化和社会媒体语境下的TIMT性能，我们推出了XHSPost，这是首个专门针对社会媒体TIMT基准测试的设计。 我们的MT$^{3}$-7B-Zero模型在最新的MIT-10M内部数据集上取得了当前最佳的结果，并且超越了包括Qwen2.5-VL-72B和InternVL2.5-78B在内的强劲基线模型，跨多个指标都有显著优势。此外，该模型还展示了对分布外语言对和数据集的强大泛化能力。 深入分析表明，多任务协同作用、强化学习初始化、课程设计以及奖励构成都为推动MLLM驱动的TIMT的发展做出了贡献。
URL
https://arxiv.org/abs/2505.19714
