Optical_Character
-
Exploration of Deep Learning Based Recognition for Urdu Text
2025-08-18 10:27:48Sumaiya Fazal, Sheeraz AhmedAbstract
Urdu is a cursive script language and has similarities with Arabic and many other South Asian languages. Urdu is difficult to classify due to its complex geometrical and morphological structure. Character classification can be processed further if segmentation technique is efficient, but due to context sensitivity in Urdu, segmentation-based recognition often results with high error rate. Our proposed approach for Urdu optical character recognition system is a component-based classification relying on automatic feature learning technique called convolutional neural network. CNN is trained and tested on Urdu text dataset, which is generated through permutation process of three characters and further proceeds to discarding unnecessary images by applying connected component technique in order to obtain ligature only. Hierarchical neural network is implemented with two levels to deal with three degrees of character permutations and component classification Our model successfully achieved 0.99% for component classification.
Abstract (translated)
乌尔都语是一种连笔书写语言，与阿拉伯语及其他许多南亚语言有相似之处。由于其复杂的几何结构和形态特征，乌尔都语难以分类。如果分割技术高效，则可以进一步处理字符分类，但由于乌尔都语的上下文敏感性，基于分割的方法通常会导致较高的错误率。我们提出的乌尔都语光学字符识别系统方法是一种基于组件的分类法，依赖于自动特征学习技术——卷积神经网络（CNN）。该 CNN 在通过三个字符排列生成的乌尔都语文本数据集上进行训练和测试，并进一步采用连通成分技术丢弃不必要的图像以获取连字。我们实施了具有两个级别的分层神经网络来处理三种字符排列程度及组件分类。我们的模型在组件分类方面成功达到了 0.99% 的精度。
URL
https://arxiv.org/abs/2508.13245
PDF
-
DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model
2025-08-18 03:28:57Qian Chen, Xianyin Zhang, Lifan Guo, Feng Chen, Chi ZhangAbstract
Recent advances in large vision-language models (LVLMs) have enabled a new paradigm of end-to-end document image parsing, excelling in Optical Character Recognition (OCR) tasks such as text, table, and formula recognition. However, generative LVLMs, similarly to large language models (LLMs), are prone to hallucinations--generating words that do not exist in input images. Furthermore, LVLMs are designed for general purposes and tend to be less effective on OCR tasks compared to expert models that are trained on domain-specific datasets. In this paper, we propose DianJin-OCR-R1, a reasoning-enhanced framework designed to address these limitations through training reasoning-and-tool interleaved VLMs. Given a recognition instruction, our DianJin-OCR-R1 model first recognizes the content in the input image by its own OCR capabilities, and then calls other tools (i.e., other expert models) to obtain their results as references, finally looks again the image and rethinks about the reasoning process to provide the final recognized content. Since architectures of expert models are tailored for specific OCR tasks, which makes them less prone to hallucinations, their results can help VLMs mitigate hallucinations. Additionally, expert models are typically smaller in scale and easy to iterate, enabling performance improvements for VLMs at a lower cost. We evaluate our model on ReST and OmniDocBench, and experimental results show that our DianJin-OCR-R1 models consistently outperform their non-reasoning counterparts and expert OCR models, which proves the effectiveness of our method.
Abstract (translated)
近期在大型视觉语言模型（LVLM）方面的进展，已经开启了端到端文档图像解析的新范式，在光学字符识别（OCR）任务如文本、表格和公式识别方面表现出色。然而，生成式的LVLM与大型语言模型（LLM）一样容易产生幻觉——即生成那些不存在于输入图像中的词语。此外，LVLM由于是为通用目的设计的，所以在OCR任务上的表现往往不如专门针对特定领域数据集训练的专业模型。 在本文中，我们提出了DianJin-OCR-R1框架，这是一个增强推理能力的设计方案，旨在通过培训具备推理与工具交织功能的视觉语言模型来解决上述限制。给定识别指令后，我们的DianJin-OCR-R1模型首先利用自身的OCR能力识别输入图像中的内容，然后调用其他专业模型（即工具）获取参考结果，并最终再次审视图像以重新思考整个推理过程，从而提供最后的内容识别结果。 由于专业模型的架构是专门针对特定OCR任务定制设计的，因此它们产生幻觉的可能性较低。这些模型的结果可以帮助LVLM减少幻觉现象的发生。此外，专业模型通常规模较小且易于迭代，这使得在不增加成本的情况下提高LVLM性能成为可能。 我们在ReST和OmniDocBench数据集上对我们的模型进行了评估，实验结果表明，DianJin-OCR-R1模型在与非推理版本的比较中以及与专业OCR模型相比均表现出更优的效果，证明了我们方法的有效性。
URL
https://arxiv.org/abs/2508.13238
PDF
-
Improving OCR for Historical Texts of Multiple Languages
2025-08-14 05:52:14Hylke Westerdijk, Ben Blankenborg, Khondoker Ittehadul IslamAbstract
This paper presents our methodology and findings from three tasks across Optical Character Recognition (OCR) and Document Layout Analysis using advanced deep learning techniques. First, for the historical Hebrew fragments of the Dead Sea Scrolls, we enhanced our dataset through extensive data augmentation and employed the Kraken and TrOCR models to improve character recognition. In our analysis of 16th to 18th-century meeting resolutions task, we utilized a Convolutional Recurrent Neural Network (CRNN) that integrated DeepLabV3+ for semantic segmentation with a Bidirectional LSTM, incorporating confidence-based pseudolabeling to refine our model. Finally, for modern English handwriting recognition task, we applied a CRNN with a ResNet34 encoder, trained using the Connectionist Temporal Classification (CTC) loss function to effectively capture sequential dependencies. This report offers valuable insights and suggests potential directions for future research.
Abstract (translated)
本文介绍了我们在光学字符识别（OCR）和文档布局分析三个任务中采用的先进深度学习技术的方法论及发现。首先，对于来自死海古卷的历史希伯来文碎片，我们通过大量数据增强扩展了我们的数据集，并使用Kraken和TrOCR模型提高了字符识别精度。在16至18世纪会议决议文本分析的任务中，我们运用了一种结合了DeepLabV3+语义分割与双向长短期记忆网络（BiLSTM）的卷积递归神经网络（CRNN），并通过基于置信度的伪标签来优化模型。最后，在现代英文手写识别任务中，我们采用了一个带有ResNet34编码器的CRNN，并通过连接主义时间分类（CTC）损失函数进行训练，以有效捕捉序列依赖关系。 本文提供了有价值的见解，并为未来的研究方向提出了建议。
URL
https://arxiv.org/abs/2508.10356
PDF
-
iWatchRoad: Scalable Detection and Geospatial Visualization of Potholes for Smart Cities
2025-08-13 15:26:03Rishi Raj Sahoo, Surbhi Saswati Mohanty, Subhankar MishraAbstract
Potholes on the roads are a serious hazard and maintenance burden. This poses a significant threat to road safety and vehicle longevity, especially on the diverse and under-maintained roads of India. In this paper, we present a complete end-to-end system called iWatchRoad for automated pothole detection, Global Positioning System (GPS) tagging, and real time mapping using OpenStreetMap (OSM). We curated a large, self-annotated dataset of over 7,000 frames captured across various road types, lighting conditions, and weather scenarios unique to Indian environments, leveraging dashcam footage. This dataset is used to fine-tune, Ultralytics You Only Look Once (YOLO) model to perform real time pothole detection, while a custom Optical Character Recognition (OCR) module was employed to extract timestamps directly from video frames. The timestamps are synchronized with GPS logs to geotag each detected potholes accurately. The processed data includes the potholes' details and frames as metadata is stored in a database and visualized via a user friendly web interface using OSM. iWatchRoad not only improves detection accuracy under challenging conditions but also provides government compatible outputs for road assessment and maintenance planning through the metadata visible on the website. Our solution is cost effective, hardware efficient, and scalable, offering a practical tool for urban and rural road management in developing regions, making the system automated. iWatchRoad is available at this https URL
Abstract (translated)
道路坑洞是严重的安全隐患和维护负担，尤其是在印度这样的多样化且维护不足的道路环境中，这对道路安全和车辆寿命构成了重大威胁。在本文中，我们介绍了一种名为iWatchRoad的端到端系统，该系统用于自动检测路面坑洞、GPS定位以及使用开放街道地图（OSM）进行实时映射。 为了开发这套系统，我们收集了一个庞大的自标注数据集，包含超过7,000帧图像，这些图像是在各种道路类型、照明条件和天气场景下捕捉到的。该数据集中使用的视频片段来源于行车记录仪。通过这一数据集，我们对Ultralytics公司的You Only Look Once (YOLO)模型进行了微调，以实现实时检测路面坑洞的功能。同时，我们还开发了一个定制化的光学字符识别（OCR）模块，用于直接从视频帧中提取时间戳信息，并将其与GPS记录同步，以便准确地标记每个发现的坑洞的位置。 处理后的数据包括坑洞详情和图像元数据，并存储在数据库中并通过友好的网络界面使用OSM进行可视化展示。iWatchRoad不仅提高了复杂条件下的检测准确性，而且还提供了政府兼容的输出用于道路评估和维护规划。我们的解决方案成本低廉、硬件效率高且可扩展，为发展中国家的城市和农村道路管理提供了一个实用工具。 此项目的详情可在以下链接中查看：[此处插入URL]
URL
https://arxiv.org/abs/2508.10945
PDF
-
Training Kindai OCR with parallel textline images and self-attention feature distance-based loss
2025-08-12 01:01:32Anh Le, Asanobu KitamotoAbstract
Kindai documents, written in modern Japanese from the late 19th to early 20th century, hold significant historical value for researchers studying societal structures, daily life, and environmental conditions of that period. However, transcribing these documents remains a labor-intensive and time-consuming task, resulting in limited annotated data for training optical character recognition (OCR) systems. This research addresses this challenge of data scarcity by leveraging parallel textline images - pairs of original Kindai text and their counterparts in contemporary Japanese fonts - to augment training datasets. We introduce a distance-based objective function that minimizes the gap between self-attention features of the parallel image pairs. Specifically, we explore Euclidean distance and Maximum Mean Discrepancy (MMD) as domain adaptation metrics. Experimental results demonstrate that our method reduces the character error rate (CER) by 2.23% and 3.94% over a Transformer-based OCR baseline when using Euclidean distance and MMD, respectively. Furthermore, our approach improves the discriminative quality of self-attention representations, leading to more effective OCR performance for historical documents.
Abstract (translated)
翻译成中文： 近代文献，即19世纪晚期至20世纪初期用现代日语编写的文件，在研究当时的社会结构、日常生活和环境条件方面具有重要的历史价值。然而，将这些文档转录仍然是一个耗时且费力的任务，导致用于训练光学字符识别（OCR）系统的注释数据有限。本研究通过利用平行文本行图像——即原始Kindai文本及其当代日文字体的对应版本——来解决这一数据稀缺问题，以扩充训练数据集。我们引入了一个基于距离的目标函数，该函数可以最小化并行图像对自注意力特征之间的差距。具体来说，我们将欧几里得距离和最大均值差异（MMD）作为领域适应度量进行研究。实验结果表明，与使用欧几里得距离的Transformer基础OCR相比，我们的方法将字符错误率降低了2.23%，而使用MMD则降低了3.94%。此外，我们的方法提高了自注意力表示的判别质量，从而为历史文献提供了更有效的OCR性能。
URL
https://arxiv.org/abs/2508.08537
PDF
-
Generating Synthetic Invoices via Layout-Preserving Content Replacement
2025-08-04 06:19:34Bevin V, Ananthakrishnan P V, Ragesh KR, Sanjay M, Vineeth S, Bibin WilsonAbstract
The performance of machine learning models for automated invoice processing is critically dependent on large-scale, diverse datasets. However, the acquisition of such datasets is often constrained by privacy regulations and the high cost of manual annotation. To address this, we present a novel pipeline for generating high-fidelity, synthetic invoice documents and their corresponding structured data. Our method first utilizes Optical Character Recognition (OCR) to extract the text content and precise spatial layout from a source invoice. Select data fields are then replaced with contextually realistic, synthetic content generated by a large language model (LLM). Finally, we employ an inpainting technique to erase the original text from the image and render the new, synthetic text in its place, preserving the exact layout and font characteristics. This process yields a pair of outputs: a visually realistic new invoice image and a perfectly aligned structured data file (JSON) reflecting the synthetic content. Our approach provides a scalable and automated solution to amplify small, private datasets, enabling the creation of large, varied corpora for training more robust and accurate document intelligence models.
Abstract (translated)
机器学习模型在自动化发票处理中的性能严重依赖于大规模、多样化的数据集。然而，由于隐私法规的限制和人工标注高昂的成本，获取这样的数据集变得非常困难。为了解决这个问题，我们提出了一种新颖的数据生成流程，用于创建高保真度的合成发票文档及其对应的结构化数据。 我们的方法首先利用光学字符识别（OCR）技术从原始发票中提取文本内容和精确的空间布局信息。然后，选择性地用大型语言模型（LLM）生成的情境真实性的合成内容替换部分数据字段。最后，我们使用一种图像修复技术去除原图中的文字，并在相同的位置渲染新的合成文字，保留了原有的版式和字体特征。 这一过程会产生一对输出：一张视觉上逼真的新发票图片以及一个与其合成内容完全对齐的结构化数据文件（JSON）。我们的方法提供了一种可扩展且自动化的解决方案来放大小规模、私有的数据集，从而能够创建大规模、多样化的语料库，用于训练更加强大和准确的文档智能模型。
URL
https://arxiv.org/abs/2508.03754
PDF
-
DocTron-Formula: Generalized Formula Recognition in Complex and Structured Scenarios
2025-08-01 04:34:17Yufeng Zhong, Zhixiong Zeng, Lei Chen, Longrong Yang, Liming Zheng, Jing Huang, Siqi Yang, Lin MaAbstract
Optical Character Recognition (OCR) for mathematical formula is essential for the intelligent analysis of scientific literature. However, both task-specific and general vision-language models often struggle to handle the structural diversity, complexity, and real-world variability inherent in mathematical content. In this work, we present DocTron-Formula, a unified framework built upon general vision-language models, thereby eliminating the need for specialized architectures. Furthermore, we introduce CSFormula, a large-scale and challenging dataset that encompasses multidisciplinary and structurally complex formulas at the line, paragraph, and page levels. Through straightforward supervised fine-tuning, our approach achieves state-of-the-art performance across a variety of styles, scientific domains, and complex layouts. Experimental results demonstrate that our method not only surpasses specialized models in terms of accuracy and robustness, but also establishes a new paradigm for the automated understanding of complex scientific documents.
Abstract (translated)
光学字符识别（OCR）对于数学公式的智能分析至关重要，这对于科学文献的智能分析尤为重要。然而，无论是特定任务的模型还是通用视觉-语言模型，在处理数学内容固有的结构多样性、复杂性和现实世界变异性方面往往面临挑战。在本研究中，我们提出了DocTron-Formula，这是一个基于通用视觉-语言模型构建的统一框架，从而消除了对专门架构的需求。此外，我们引入了CSFormula，这是包含多学科和结构复杂的公式的大规模且具有挑战性的数据集，这些公式涵盖了行、段落及页面级别。通过简单的监督微调，我们的方法在各种风格、科学领域以及复杂布局方面均实现了最先进的性能。实验结果表明，我们的方法不仅在准确性和鲁棒性上超越了专门模型，并且为自动理解复杂的科学文档建立了一个新的范式。
URL
https://arxiv.org/abs/2508.00311
PDF
-
Medical Image De-Identification Benchmark Challenge
2025-07-31 14:47:20Linmin Pei, Granger Sutton, Michael Rutherford, Ulrike Wagner, Tracy Nolan, Kirk Smith, Phillip Farmer, Peter Gu, Ambar Rana, Kailing Chen, Thomas Ferleman, Brian Park, Ye Wu, Jordan Kojouharov, Gargi Singh, Jon Lemon, Tyler Willis, Milos Vukadinovic, Grant Duffy, Bryan He, David Ouyang, Marco Pereanez, Daniel Samber, Derek A. Smith, Christopher Cannistraci, Zahi Fayad, David S. Mendelson, Michele Bufano, Elmar Kotter, Hamideh Haghiri, Rajesh Baidya, Stefan Dvoretskii, Klaus H. Maier-Hein, Marco Nolden, Christopher Ablett, Silvia Siggillino, Sandeep Kaushik, Hongzhu Jiang, Sihan Xie, Zhiyu Wan, Alex Michie, Simon J Doran, Angeline Aurelia Waly, Felix A. Nathaniel Liang, Humam Arshad Mustagfirin, Michelle Grace Felicia, Kuo Po Chih, Rahul Krish, Ghulam Rasool, Nidhal Bouaynaya, Nikolas Koutsoubis, Kyle Naddeo, Kartik Pandit, Tony O'Sullivan, Raj Krish, Qinyan Pan, Scott Gustafson, Benjamin Kopchick, Laura Opsahl-Ong, Andrea Olvera-Morales, Jonathan Pinney, Kathryn Johnson, Theresa Do, Juergen Klenk, Maria Diaz, Arti Singh, Rong Chai, David A. Clunie, Fred Prior, Keyvan FarahaniAbstract
The de-identification (deID) of protected health information (PHI) and personally identifiable information (PII) is a fundamental requirement for sharing medical images, particularly through public repositories, to ensure compliance with patient privacy laws. In addition, preservation of non-PHI metadata to inform and enable downstream development of imaging artificial intelligence (AI) is an important consideration in biomedical research. The goal of MIDI-B was to provide a standardized platform for benchmarking of DICOM image deID tools based on a set of rules conformant to the HIPAA Safe Harbor regulation, the DICOM Attribute Confidentiality Profiles, and best practices in preservation of research-critical metadata, as defined by The Cancer Imaging Archive (TCIA). The challenge employed a large, diverse, multi-center, and multi-modality set of real de-identified radiology images with synthetic PHI/PII inserted. The MIDI-B Challenge consisted of three phases: training, validation, and test. Eighty individuals registered for the challenge. In the training phase, we encouraged participants to tune their algorithms using their in-house or public data. The validation and test phases utilized the DICOM images containing synthetic identifiers (of 216 and 322 subjects, respectively). Ten teams successfully completed the test phase of the challenge. To measure success of a rule-based approach to image deID, scores were computed as the percentage of correct actions from the total number of required actions. The scores ranged from 97.91% to 99.93%. Participants employed a variety of open-source and proprietary tools with customized configurations, large language models, and optical character recognition (OCR). In this paper we provide a comprehensive report on the MIDI-B Challenge's design, implementation, results, and lessons learned.
Abstract (translated)
保护健康信息（PHI）和个人识别信息（PII）的去标识化是分享医学影像资料，特别是通过公共存储库进行分享的基本要求，以确保符合患者隐私法的规定。此外，在生物医学研究中保留非PHI元数据，以便促进和助力下游成像人工智能（AI）的发展，也是重要的考量因素之一。 MIDI-B项目的目的是提供一个标准化的平台，用于根据符合HIPAA安全港法规、DICOM属性保密配置文件以及由癌症影像档案馆（TCIA）定义的研究关键元数据保存最佳实践的一套规则来对DICOM图像去标识化工具进行基准测试。挑战赛采用了大量多中心和多种模态的真实放射学脱敏图像，并插入了合成的PHI/PII。 MIDI-B挑战赛包括三个阶段：训练、验证和测试。共有80名参与者注册参加了该挑战赛。在训练阶段，我们鼓励参赛者使用他们自己的内部数据或公开数据来调整他们的算法。验证和测试阶段分别利用包含216名和322名患者合成标识符的DICOM图像进行。 十支队伍成功完成了挑战赛的测试阶段。为了衡量基于规则的方法在影像去标识化中的效果，得分计算为正确操作数占总需执行操作数的百分比。分数范围从97.91%到99.93%不等。参赛者采用了各种开源和专有工具（带有定制配置）、大型语言模型以及光学字符识别（OCR）技术。 本文提供了关于MIDI-B挑战赛的设计、实施、结果及经验教训的全面报告。
URL
https://arxiv.org/abs/2507.23608
PDF
-
Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approach
2025-07-31 03:23:04Yanming Xiu, Maria GorlatovaAbstract
The virtual content in augmented reality (AR) can introduce misleading or harmful information, leading to semantic misunderstandings or user errors. In this work, we focus on visual information manipulation (VIM) attacks in AR where virtual content changes the meaning of real-world scenes in subtle but impactful ways. We introduce a taxonomy that categorizes these attacks into three formats: character, phrase, and pattern manipulation, and three purposes: information replacement, information obfuscation, and extra wrong information. Based on the taxonomy, we construct a dataset, AR-VIM. It consists of 452 raw-AR video pairs spanning 202 different scenes, each simulating a real-world AR scenario. To detect such attacks, we propose a multimodal semantic reasoning framework, VIM-Sense. It combines the language and visual understanding capabilities of vision-language models (VLMs) with optical character recognition (OCR)-based textual analysis. VIM-Sense achieves an attack detection accuracy of 88.94% on AR-VIM, consistently outperforming vision-only and text-only baselines. The system reaches an average attack detection latency of 7.07 seconds in a simulated video processing framework and 7.17 seconds in a real-world evaluation conducted on a mobile Android AR application.
Abstract (translated)
在增强现实（AR）中，虚拟内容可能会引入误导性或有害的信息，导致语义误解或用户错误。在这项工作中，我们关注的是视觉信息操纵（VIM）攻击，在这种攻击中，虚拟内容以微妙但有影响力的方式改变了真实场景的意义。我们提出了一个分类法，将这些攻击分为三种格式：字符、短语和模式操作，以及三个目的：信息替换、信息混淆和额外错误信息。 基于这一分类法，我们构建了一个数据集AR-VIM，其中包括452对原始AR视频片段，涵盖了202种不同的场景，每一种都模拟了现实中的AR情景。为了检测此类攻击，我们提出了一种多模态语义推理框架VIM-Sense，该框架结合了视觉-语言模型（VLM）的语言和视觉理解能力以及基于光学字符识别（OCR）的文本分析。 在AR-VIM数据集上，VIM-Sense实现了88.94%的攻击检测准确率，并且始终优于仅使用视觉或仅使用文本的基础方法。该系统在一个模拟视频处理框架中达到平均7.07秒的攻击检测延迟，在移动Android AR应用程序的真实世界评估中达到了平均7.17秒的延迟。
URL
https://arxiv.org/abs/2507.20356
PDF
-
Zero-shot OCR Accuracy of Low-Resourced Languages: A Comparative Analysis on Sinhala and Tamil
2025-07-24 10:08:43Nevidu Jayatilleke, Nisansa de SilvaAbstract
Solving the problem of Optical Character Recognition (OCR) on printed text for Latin and its derivative scripts can now be considered settled due to the volumes of research done on English and other High-Resourced Languages (HRL). However, for Low-Resourced Languages (LRL) that use unique scripts, it remains an open problem. This study presents a comparative analysis of the zero-shot performance of six distinct OCR engines on two LRLs: Sinhala and Tamil. The selected engines include both commercial and open-source systems, aiming to evaluate the strengths of each category. The Cloud Vision API, Surya, Document AI, and Tesseract were evaluated for both Sinhala and Tamil, while Subasa OCR and EasyOCR were examined for only one language due to their limitations. The performance of these systems was rigorously analysed using five measurement techniques to assess accuracy at both the character and word levels. According to the findings, Surya delivered the best performance for Sinhala across all metrics, with a WER of 2.61%. Conversely, Document AI excelled across all metrics for Tamil, highlighted by a very low CER of 0.78%. In addition to the above analysis, we also introduce a novel synthetic Tamil OCR benchmarking dataset.
Abstract (translated)
解决拉丁语及其衍生脚本印刷文本的光学字符识别（OCR）问题，由于对英语及其他高资源语言（HRL）进行了大量的研究工作，现在可以被认为是基本解决了。然而，对于使用独特脚本的低资源语言（LRL），这一问题仍然存在挑战性。这项研究通过对斯里兰卡语和泰米尔语这两种LRL进行比较分析，探讨了六种不同OCR引擎在零样本设置下的性能表现。所选的系统包括商业和开源两类，旨在评估每类系统的优缺点。对于斯里兰卡语和泰米尔语，分别测试了Cloud Vision API、Surya、Document AI以及Tesseract这四个系统；而由于Subasa OCR和EasyOCR的局限性，仅对其中一个语言进行了评测。通过五种不同的度量方法来评估这些系统的字符级和词级准确率，并对其性能进行了严格分析。 研究结果表明，在所有指标中，Surya在斯里兰卡语上表现最佳，其词错误率为2.61%；而在泰米尔语方面，Document AI则在所有测量标准下均表现出色，尤其值得一提的是其字符错误率仅为0.78%。此外，除了上述分析外，我们还引入了一套全新的合成泰米尔语OCR基准数据集。
URL
https://arxiv.org/abs/2507.18264
PDF
-
Aesthetics is Cheap, Show me the Text: An Empirical Evaluation of State-of-the-Art Generative Models for OCR
2025-07-20 18:43:09Peirong Zhang, Haowei Xu, Jiaxin Zhang, Guitao Xu, Xuhan Zheng, Zhenhua Yang, Junle Liu, Yuyi Zhang, Lianwen JinAbstract
Text image is a unique and crucial information medium that integrates visual aesthetics and linguistic semantics in modern e-society. Due to their subtlety and complexity, the generation of text images represents a challenging and evolving frontier in the image generation field. The recent surge of specialized image generators (\emph{e.g.}, Flux-series) and unified generative models (\emph{e.g.}, GPT-4o), which demonstrate exceptional fidelity, raises a natural question: can they master the intricacies of text image generation and editing? Motivated by this, we assess current state-of-the-art generative models' capabilities in terms of text image generation and editing. We incorporate various typical optical character recognition (OCR) tasks into our evaluation and broaden the concept of text-based generation tasks into OCR generative tasks. We select 33 representative tasks and categorize them into five categories: document, handwritten text, scene text, artistic text, and complex \& layout-rich text. For comprehensive evaluation, we examine six models across both closed-source and open-source domains, using tailored, high-quality image inputs and prompts. Through this evaluation, we draw crucial observations and identify the weaknesses of current generative models for OCR tasks. We argue that photorealistic text image generation and editing should be internalized as foundational skills into general-domain generative models, rather than being delegated to specialized solutions, and we hope this empirical analysis can provide valuable insights for the community to achieve this goal. This evaluation is online and will be continuously updated at our GitHub repository.
Abstract (translated)
文本图像是一种在现代电子社会中集视觉美感与语言语义于一体的独特且关键的信息媒介。由于其微妙性和复杂性，生成文本图像是图像生成领域中的一个充满挑战和不断发展的前沿方向。近年来，专业化的图像生成器（如Flux系列）以及统一的生成模型（如GPT-4o），展现出非凡的真实感效果，这引发了一个自然的问题：它们能否掌握文本图像生成与编辑的复杂性？受到这一问题的启发，我们评估了当前最先进的生成模型在文本图像生成和编辑方面的能力。我们在评估中纳入了一系列典型的光学字符识别(OCR)任务，并将基于文本的生成任务的概念扩展到了OCR生成任务。我们选择了33个具有代表性的任务，并将其分类为五类：文档、手写文本、场景文本、艺术文本以及复杂且布局丰富的文本。为了进行全面评估，我们在闭源和开源领域中选取了六种模型，并使用定制的高质量图像输入与提示语进行测试。通过这一评价过程，我们得出了关键性的观察结果并识别出当前OCR任务中的生成模型存在的弱点。我们认为，逼真的文本图像生成和编辑应当被融入到通用领域的生成模型之中，而不是委派给专门的解决方案来处理，并希望这种实证分析能够为社区实现此目标提供宝贵的见解。此次评估是在线进行且将持续在我们的GitHub仓库中更新。 这段文字介绍了对当前最先进的生成模型在文本图像生成和编辑方面的能力进行了全面评估的研究工作。研究不仅涵盖了多种类型的OCR任务，还提出了将逼真的文本图像生成和编辑技术融入通用领域生成模型的重要性，并分享了持续更新的实证分析结果。
URL
https://arxiv.org/abs/2507.15085
PDF
-
Seeing the Signs: A Survey of Edge-Deployable OCR Models for Billboard Visibility Analysis
2025-07-15 20:58:24Maciej Szankin, Vidhyananth Venkatasamy, Lihang YingAbstract
Outdoor advertisements remain a critical medium for modern marketing, yet accurately verifying billboard text visibility under real-world conditions is still challenging. Traditional Optical Character Recognition (OCR) pipelines excel at cropped text recognition but often struggle with complex outdoor scenes, varying fonts, and weather-induced visual noise. Recently, multimodal Vision-Language Models (VLMs) have emerged as promising alternatives, offering end-to-end scene understanding with no explicit detection step. This work systematically benchmarks representative VLMs - including Qwen 2.5 VL 3B, InternVL3, and SmolVLM2 - against a compact CNN-based OCR baseline (PaddleOCRv4) across two public datasets (ICDAR 2015 and SVT), augmented with synthetic weather distortions to simulate realistic degradation. Our results reveal that while selected VLMs excel at holistic scene reasoning, lightweight CNN pipelines still achieve competitive accuracy for cropped text at a fraction of the computational cost-an important consideration for edge deployment. To foster future research, we release our weather-augmented benchmark and evaluation code publicly.
Abstract (translated)
户外广告仍然是现代营销中的关键媒介，但在现实条件下准确验证广告牌文本的可见性仍然具有挑战性。传统的光学字符识别（OCR）管道在处理裁剪后的文字时表现出色，但对于复杂的户外场景、多变的字体以及天气引起的视觉噪声却往往力不从心。最近，多模态视觉-语言模型（VLMs）作为有前途的替代方案出现，提供了端到端的场景理解能力，并且不需要显式的检测步骤。本研究系统地评估了具有代表性的几种VLM——包括Qwen 2.5 VL 3B、InternVL3和SmolVLM2，在两个公开数据集（ICDAR 2015和SVT）上，这些数据集通过合成天气扭曲进行了扩充以模拟现实中的退化情况。我们的结果显示，虽然所选的VLM在整体场景推理方面表现出色，但轻量级的CNN管道仍然能够实现与裁剪文本相当的准确度，并且计算成本更低——这对于边缘部署来说是一个重要的考虑因素。为了促进未来的研究，我们公开发布了带有天气增强基准和评估代码的数据集。
URL
https://arxiv.org/abs/2507.11730
PDF
-
A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends
2025-07-14 02:10:31Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Geoffrey Martin, Yifan PengAbstract
Visually-Rich Document Understanding (VRDU) has emerged as a critical field, driven by the need to automatically process documents containing complex visual, textual, and layout information. Recently, Multimodal Large Language Models (MLLMs) have shown remarkable potential in this domain, leveraging both Optical Character Recognition (OCR)-dependent and OCR-free frameworks to extract and interpret information in document images. This survey reviews recent advancements in MLLM-based VRDU, highlighting three core components: (1) methods for encoding and fusing textual, visual, and layout features; (2) training paradigms, including pretraining strategies, instruction-response tuning, and the trainability of different model modules; and (3) datasets utilized for pretraining, instruction-tuning, and supervised fine-tuning. Finally, we discuss the challenges and opportunities in this evolving field and propose future directions to advance the efficiency, generalizability, and robustness of VRDU systems.
Abstract (translated)
视觉丰富文档理解（VRDU）作为一项关键领域已经出现，这一领域的兴起源于自动处理包含复杂视觉、文本和布局信息的文档的需求。最近，多模态大型语言模型（MLLMs）在该领域展现了显著潜力，通过依赖光学字符识别（OCR）和不依赖OCR的框架来提取并解释文档图像中的信息。这篇综述回顾了基于MLLM的VRDU领域的近期进展，并重点介绍了三个核心组成部分：(1) 文本、视觉和布局特征编码及融合的方法；(2) 训练范式，包括预训练策略、指令-响应调优以及不同模型模块的可训练性；(3) 用于预训练、指令调优和监督微调的数据集。最后，我们讨论了这一不断发展的领域面临的挑战与机遇，并提出未来推进VRDU系统效率、泛化能力和鲁棒性的方向。
URL
https://arxiv.org/abs/2507.09861
PDF
-
Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
2025-07-11 05:02:06Yupu Liang, Yaping Zhang, Zhiyang Zhang, Zhiyuan Chen, Yang Zhao, Lu Xiang, Chengqing Zong, Yu ZhouAbstract
Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.
Abstract (translated)
多模态大型语言模型（MLLMs）在文档图像任务中表现出色，尤其是在光学字符识别（OCR）方面。然而，它们在处理文档图像机器翻译（DIMT）时遇到困难，因为这需要同时应对跨模式和跨语言的挑战。之前通过在DIMT数据集上进行监督微调(SFT)来增强DIMT能力的方法往往会遗忘模型原有的单语技能，如OCR。为了解决这些挑战，我们引入了一种新的微调范式——同步自审（SSR），旨在保持其OCR专业技能，灵感来自“双语认知优势”这一概念。具体来说，SSR提示模型在生成翻译文本之前先生成OCR文本，这使得模型能够利用其强大的单语言OCR能力同时学习跨语言的文本翻译。全面的实验表明，所提出的SSR学习有助于缓解灾难性遗忘，从而提高MLLMs在OCR和DIMT任务上的泛化能力。
URL
https://arxiv.org/abs/2507.08309
PDF
-
Evaluating the Impact of Khmer Font Types on Text Recognition
2025-06-30 15:35:51Vannkinh Nom, Souhail Bakkali, Muhammad Muzzamil Luqman, Mickael Coustaty, Jean-Marc OgierAbstract
Text recognition is significantly influenced by font types, especially for complex scripts like Khmer. The variety of Khmer fonts, each with its unique character structure, presents challenges for optical character recognition (OCR) systems. In this study, we evaluate the impact of 19 randomly selected Khmer font types on text recognition accuracy using Pytesseract. The fonts include Angkor, Battambang, Bayon, Bokor, Chenla, Dangrek, Freehand, Kh Kompong Chhnang, Kh SN Kampongsom, Khmer, Khmer CN Stueng Songke, Khmer Savuth Pen, Metal, Moul, Odor MeanChey, Preah Vihear, Siemreap, Sithi Manuss, and iSeth First. Our comparison of OCR performance across these fonts reveals that Khmer, Odor MeanChey, Siemreap, Sithi Manuss, and Battambang achieve high accuracy, while iSeth First, Bayon, and Dangrek perform poorly. This study underscores the critical importance of font selection in optimizing Khmer text recognition and provides valuable insights for developing more robust OCR systems.
Abstract (translated)
文本识别在很大程度上受到字体类型的影响，尤其是在处理像高棉语这样复杂的脚本时。由于每种高棉语字体都有其独特的字符结构，这给光学字符识别（OCR）系统带来了挑战。在这项研究中，我们使用Pytesseract评估了19种随机选择的高棉语字体对文本识别准确率的影响。这些字体包括Angkor、Battambang、Bayon、Bokor、Chenla、Dangrek、Freehand、Kh Kompong Chhnang、Kh SN Kampongsom、Khmer、Khmer CN Stueng Songke、Khmer Savuth Pen、Metal、Moul、Odor MeanChey、Preah Vihear、Siemreap、Sithi Manuss和iSeth First。我们对这些字体的OCR性能进行了比较，结果显示Khmer、Odor MeanChey、Siemreap、Sithi Manuss以及Battambang字体实现了较高的准确率，而iSeth First、Bayon和Dangrek的表现较差。这项研究表明，在优化高棉语文本识别中选择合适的字体至关重要，并为开发更强大的OCR系统提供了宝贵的见解。
URL
https://arxiv.org/abs/2506.23963
PDF
-
SPICE-HL3: Single-Photon, Inertial, and Stereo Camera dataset for Exploration of High-Latitude Lunar Landscapes
2025-06-28 17:21:36David Rodr\'iguez-Mart\'inez, Dave van der Meer, Junlin Song, Abishek Bera, C. J. P\'erez-del-Pulgar, Miguel Angel Olivares-MendezAbstract
Exploring high-latitude lunar regions presents an extremely challenging visual environment for robots. The low sunlight elevation angle and minimal light scattering result in a visual field dominated by a high dynamic range featuring long, dynamic shadows. Reproducing these conditions on Earth requires sophisticated simulators and specialized facilities. We introduce a unique dataset recorded at the LunaLab from the SnT - University of Luxembourg, an indoor test facility designed to replicate the optical characteristics of multiple lunar latitudes. Our dataset includes images, inertial measurements, and wheel odometry data from robots navigating seven distinct trajectories under multiple illumination scenarios, simulating high-latitude lunar conditions from dawn to night time with and without the aid of headlights, resulting in 88 distinct sequences containing a total of 1.3M images. Data was captured using a stereo RGB-inertial sensor, a monocular monochrome camera, and for the first time, a novel single-photon avalanche diode (SPAD) camera. We recorded both static and dynamic image sequences, with robots navigating at slow (5 cm/s) and fast (50 cm/s) speeds. All data is calibrated, synchronized, and timestamped, providing a valuable resource for validating perception tasks from vision-based autonomous navigation to scientific imaging for future lunar missions targeting high-latitude regions or those intended for robots operating across perceptually degraded environments. The dataset can be downloaded from this https URL, and a visual overview is available at this https URL. All supplementary material can be found at this https URL.
Abstract (translated)
探索高纬度的月球区域对机器人来说是一个极具挑战性的视觉环境。由于低太阳高度角和少量光线散射，这些地区的视野中充满了宽广的动态阴影，导致光照范围极大。在地球上再现这种条件需要复杂的模拟器和专门设施。 我们介绍了一套独特的数据集，该数据集由卢森堡大学SnT的研究机构LunaLab室内测试设施记录而成。这个实验室旨在复制不同月球纬度的光学特性。我们的数据集包含了机器人在七条不同的轨迹上，在多种光照条件下导航时所拍摄的图像、惯性测量值和轮式里程计数据，模拟了从黎明到夜晚的不同高纬度月球条件，包括开启和关闭前照灯的情况，总共包含88种不同的序列，总共有130万张图片。这些数据使用了一套立体RGB惯性传感器、单色相机以及首次使用的新型单光子雪崩二极管（SPAD）摄像头进行采集。 我们记录了静态和动态的图像序列，在慢速（5厘米/秒）和快速（50厘米/秒）的速度下，机器人在导航过程中的表现。所有数据都经过校准、同步并添加时间戳，为从基于视觉的自主导航任务到未来针对高纬度地区或感知环境恶劣条件下的机器人的科学成像等验证提供了宝贵的资源。 该数据集可以从以下链接下载：[数据集下载链接]（请将方括号内的文本替换为实际URL）。 可视化概览可以在以下链接查看：[视觉概览链接] （请将方括号内的文本替换为实际URL）。 所有补充材料可以在以下链接找到：[补充材料链接](请将方括号内的文本替换为实际URL)。
URL
https://arxiv.org/abs/2506.22956
PDF
-
Multimodal Misinformation Detection Using Early Fusion of Linguistic, Visual, and Social Features
2025-06-26 18:17:35Gautam Kishore ShahiAbstract
Amid a tidal wave of misinformation flooding social media during elections and crises, extensive research has been conducted on misinformation detection, primarily focusing on text-based or image-based approaches. However, only a few studies have explored multimodal feature combinations, such as integrating text and images for building a classification model to detect misinformation. This study investigates the effectiveness of different multimodal feature combinations, incorporating text, images, and social features using an early fusion approach for the classification model. This study analyzed 1,529 tweets containing both text and images during the COVID-19 pandemic and election periods collected from Twitter (now X). A data enrichment process was applied to extract additional social features, as well as visual features, through techniques such as object detection and optical character recognition (OCR). The results show that combining unsupervised and supervised machine learning models improves classification performance by 15% compared to unimodal models and by 5% compared to bimodal models. Additionally, the study analyzes the propagation patterns of misinformation based on the characteristics of misinformation tweets and the users who disseminate them.
Abstract (translated)
在选举和危机期间，随着社交媒体上错误信息泛滥成灾，已经进行了大量关于错误信息检测的研究，主要集中在基于文本或图像的方法。然而，只有少数研究探讨了多模态特征组合的使用，例如将文本与图像相结合以构建用于检测错误信息的分类模型。这项研究调查了不同多模态特征组合的有效性，通过早期融合方法在分类模型中结合文本、图片和社会特征进行分析。该研究分析了1,529条包含文本和图片的推特，在新冠疫情期间以及选举期间从Twitter（现为X）收集的数据。 为了提取额外的社会特征及视觉特征，采用数据增强过程，并应用如对象检测和光学字符识别（OCR）等技术。结果显示，结合无监督和有监督机器学习模型能比单一模态模型提高15%的分类性能，相较于双模态模型也有5%的提升。此外，该研究还根据错误信息推文的特征及其传播者的特点分析了错误信息的传播模式。
URL
https://arxiv.org/abs/2507.01984
PDF
-
Logios : An open source Greek Polytonic Optical Character Recognition system
2025-06-26 17:04:27Perifanos Konstantinos, Goutsos DionisisAbstract
In this paper, we present an Optical Character Recognition (OCR) system specifically designed for the accurate recognition and digitization of Greek polytonic texts. By leveraging the combined strengths of convolutional layers for feature extraction and recurrent layers for sequence learning, our system addresses the unique challenges posed by Greek polytonic scripts. This approach aims to overcome the limitations of traditional OCR methods, offering significant improvements in accuracy and efficiency. We release the underlying model as an open-source library and make our OCR platform available for academic use.
Abstract (translated)
在这篇论文中，我们提出了一种专门用于准确识别和数字化希腊多音调文本的光学字符识别（OCR）系统。通过利用卷积层进行特征提取和递归层进行序列学习的综合优势，我们的系统解决了由希腊多音调脚本带来的独特挑战。这种方法旨在克服传统OCR方法的局限性，在准确性和效率方面提供了显著改进。我们将基础模型作为开源库发布，并将我们的OCR平台提供给学术界使用。
URL
https://arxiv.org/abs/2506.21474
PDF
-
Unfolding the Past: A Comprehensive Deep Learning Approach to Analyzing Incunabula Pages
2025-06-22 15:33:20Klaudia Ropel, Krzysztof Kutt, Luiz do Valle Miranda, Grzegorz J. NalepaAbstract
We developed a proof-of-concept method for the automatic analysis of the structure and content of incunabula pages. A custom dataset comprising 500 annotated pages from five different incunabula was created using resources from the Jagiellonian Digital Library. Each page was manually labeled with five predefined classes: Text, Title, Picture, Table, and Handwriting. Additionally, the publicly available DocLayNet dataset was utilized as supplementary training data. To perform object detection, YOLO11n and YOLO11s models were employed and trained using two strategies: a combined dataset (DocLayNet and the custom dataset) and the custom dataset alone. The highest performance (F1 = 0.94) was achieved by the YOLO11n model trained exclusively on the custom data. Optical character recognition was then conducted on regions classified as Text, using both Tesseract and Kraken OCR, with Tesseract demonstrating superior results. Subsequently, image classification was applied to the Picture class using a ResNet18 model, achieving an accuracy of 98.7% across five subclasses: Decorative_letter, Illustration, Other, Stamp, and Wrong_detection. Furthermore, the CLIP model was utilized to generate semantic descriptions of illustrations. The results confirm the potential of machine learning in the analysis of early printed books, while emphasizing the need for further advancements in OCR performance and visual content interpretation.
Abstract (translated)
我们开发了一种概念验证方法，用于自动分析古籍印刷品页面的结构和内容。使用来自雅盖隆数字图书馆（Jagiellonian Digital Library）的资源创建了一个包含500页注释数据集，这些注释来自于五本不同的古登堡圣经，每一页都被手动标注为五个预定义类别之一：文本、标题、图片、表格和手写。此外，还使用了公开可用的DocLayNet数据集作为辅助训练数据。 为了执行目标检测任务，我们采用了YOLO11n和YOLO11s模型，并通过两种策略对其进行训练：一种是结合使用的数据集（DocLayNet和自定义数据集），另一种则是仅使用自定义数据集。结果显示，专门在自定义数据集上进行训练的YOLO11n模型取得了最佳性能（F1分数为0.94）。 随后，我们对被分类为“文本”的区域进行了光学字符识别（OCR）。这项任务采用了Tesseract和Kraken两种OCR工具，其中Tesseract表现更优。接着，我们使用ResNet18模型对图片类别进行图像分类，在五种子类（装饰字母、插图、其他、印章以及错误检测）中实现了98.7%的准确率。最后，还利用了CLIP模型生成关于插图的语义描述。 这些结果证实了机器学习在分析早期印刷书籍方面的潜力，同时也强调了需要进一步提高OCR性能和视觉内容解读方面的发展需求。
URL
https://arxiv.org/abs/2506.18069
PDF
-
Connecting Vision and Emissions: A Behavioural AI Approach to Carbon Estimation in Road Design
2025-06-18 11:50:24Ammar K Al Mhdawi, Nonso Nnamoko, Safanah Mudheher Raafat, M. K. S. Al-Mhdawi, Amjad J HumaidiAbstract
We present an enhanced YOLOv8 real time vehicle detection and classification framework, for estimating carbon emissions in urban environments. The system enhances YOLOv8 architecture to detect, segment, and track vehicles from live traffic video streams. Once a vehicle is localized, a dedicated deep learning-based identification module is employed to recognize license plates and classify vehicle types. Since YOLOv8 lacks the built-in capacity for fine grained recognition tasks such as reading license plates or determining vehicle attributes beyond class labels, our framework incorporates a hybrid pipeline where each detected vehicle is tracked and its bounding box is cropped and passed to a deep Optical Character Recognition (OCR) module. This OCR system, composed of multiple convolutional neural network (CNN) layers, is trained specifically for character-level detection and license plate decoding under varied conditions such as motion blur, occlusion, and diverse font styles. Additionally, the recognized plate information is validated using a real time API that cross references with an external vehicle registration database to ensure accurate classification and emission estimation. This multi-stage approach enables precise, automated calculation of per vehicle carbon emissions. Extensive evaluation was conducted using a diverse vehicle dataset enriched with segmentation masks and annotated license plates. The YOLOv8 detector achieved a mean Average Precision (mAP@0.5) of approximately 71% for bounding boxes and 70% for segmentation masks. Character level OCR accuracy reached up to 99% with the best performing CNN model. These results affirm the feasibility of combining real time object detection with deep OCR for practical deployment in smart transportation systems, offering a scalable solution for automated, vehicle specific carbon emission monitoring.
Abstract (translated)
我们提出了一种增强版的YOLOv8实时车辆检测和分类框架，用于在城市环境中估算碳排放。该系统改进了YOLOv8架构，能够从实时交通视频流中检测、分割并追踪车辆。一旦定位到一辆车，将使用一个专门基于深度学习的身份识别模块来识别车牌号码，并对车辆类型进行分类。由于YOLOv8本身缺乏读取车牌或确定超出类别标签的车辆属性等精细化任务的能力，我们的框架采用了一种混合流水线方法，在该方法中，每辆检测到的车辆都会被追踪并且其边界框会被裁剪并传递给一个深度光学字符识别（OCR）模块。这个OCR系统由多个卷积神经网络(CNN)层组成，并且特别针对在运动模糊、遮挡和不同字体样式等条件下进行字符级检测和车牌解码进行了训练。 此外，通过实时API验证所识读的车牌信息，该API与外部车辆注册数据库交叉参考以确保准确分类和排放估算。这种多阶段方法能够实现每辆车碳排放的精确自动化计算。使用一组多样化的带有分割掩模和注释车牌的数据集进行广泛的评估后发现，YOLOv8检测器在边界框方面达到了约71%的平均精度(mAP@0.5)，而在分割掩模方面则为70%。字符级OCR准确率最高可达99%，这表明结合实时目标检测与深度OCR在智能交通系统中的实际部署是可行的，提供了一种针对自动化、车辆特定碳排放监测的可扩展解决方案。
URL
https://arxiv.org/abs/2506.18924
PDF
