Abstract
While recent advancements in Image Super-Resolution (SR) using diffusion models have shown promise in improving overall image quality, their application to scene text images has revealed limitations. These models often struggle with accurate text region localization and fail to effectively model image and multilingual character-to-shape priors. This leads to inconsistencies, the generation of hallucinated textures, and a decrease in the perceived quality of the super-resolved text. To address these issues, we introduce TextSR, a multimodal diffusion model specifically designed for Multilingual Scene Text Image Super-Resolution. TextSR leverages a text detector to pinpoint text regions within an image and then employs Optical Character Recognition (OCR) to extract multilingual text from these areas. The extracted text characters are then transformed into visual shapes using a UTF-8 based text encoder and cross-attention. Recognizing that OCR may sometimes produce inaccurate results in real-world scenarios, we have developed two innovative methods to enhance the robustness of our model. By integrating text character priors with the low-resolution text images, our model effectively guides the super-resolution process, enhancing fine details within the text and improving overall legibility. The superior performance of our model on both the TextZoom and TextVQA datasets sets a new benchmark for STISR, underscoring the efficacy of our approach.
Abstract (translated)
尽管最近使用扩散模型在图像超分辨率（SR）方面取得了进展，这些方法在提高整体图像质量方面显示出潜力，但它们在应用于场景文本图像时却暴露出了一些局限性。这些模型往往难以准确地定位文本区域，并且无法有效建模图像和多语言字符到形状的先验知识，这导致了不一致、生成幻觉纹理以及超分辨率后文本感知质量下降的问题。 为了应对这些问题，我们引入了一种专为多语言场景文本图像超分辨率设计的TextSR多模式扩散模型。TextSR利用一个文本检测器来识别图像中的文本区域，并使用光学字符识别（OCR）从这些区域中提取多语言文本。然后，通过基于UTF-8的文本编码和跨注意力机制将提取出的文字转换为视觉形状。 考虑到在实际场景中，OCR可能会产生不准确的结果，我们开发了两种创新的方法来增强模型的鲁棒性。通过结合低分辨率图像中的字符先验知识，我们的模型能够有效地指导超分辨率过程，提高文字细节并提升整体可读性。我们在TextZoom和TextVQA数据集上的测试结果表明，我们的方法在场景文本图像超分辨率（STISR）领域树立了新的性能标杆，这进一步证明了我们方法的有效性和优越性。
URL
https://arxiv.org/abs/2505.23119
