Abstract
Microscopic characterizations, such as Scanning Electron Microscopy (SEM), are widely used in scientific research for visualizing and analyzing microstructures. Determining the scale bars is an important first step of accurate SEM analysis; however, currently, it mainly relies on manual operations, which is both time-consuming and prone to errors. To address this issue, we propose a multi-modal and automated scale bar detection and extraction framework that provides concurrent object detection, text detection and text recognition with a Large Language Model (LLM) agent. The proposed framework operates in four phases; i) Automatic Dataset Generation (Auto-DG) model to synthesize a diverse dataset of SEM images ensuring robust training and high generalizability of the model, ii) scale bar object detection, iii) information extraction using a hybrid Optical Character Recognition (OCR) system with DenseNet and Convolutional Recurrent Neural Network (CRNN) based algorithms, iv) an LLM agent to analyze and verify accuracy of the results. The proposed model demonstrates a strong performance in object detection and accurate localization with a precision of 100%, recall of 95.8%, and a mean Average Precision (mAP) of 99.2% at IoU=0.5 and 69.1% at IoU=0.5:0.95. The hybrid OCR system achieved 89% precision, 65% recall, and a 75% F1 score on the Auto-DG dataset, significantly outperforming several mainstream standalone engines, highlighting its reliability for scientific image analysis. The LLM is introduced as a reasoning engine as well as an intelligent assistant that suggests follow-up steps and verifies the results. This automated method powered by an LLM agent significantly enhances the efficiency and accuracy of scale bar detection and extraction in SEM images, providing a valuable tool for microscopic analysis and advancing the field of scientific imaging.
Abstract (translated)
微观表征技术，如扫描电子显微镜（SEM），在科学研究中广泛用于可视化和分析微观结构。确定标尺是准确进行SEM分析的重要第一步；然而，目前主要依靠手动操作来完成这一过程，这不仅耗时而且容易出错。为了解决这个问题，我们提出了一种多模态且自动化的标尺检测与提取框架，该框架结合了对象检测、文本检测和基于大型语言模型（LLM）的文本识别功能。 提出的框架分为四个阶段：i) 自动数据生成(Auto-DG)模型，用于合成各种SEM图像的数据集，确保模型训练的稳健性和高泛化能力；ii) 标尺对象检测；iii) 使用密集网络(DenseNet)和卷积循环神经网络(CRNN)算法组成的混合光学字符识别(OCR)系统进行信息提取；iv) 一个LLM代理来分析并验证结果的准确性。 所提出的模型在物体检测方面表现出色，且精确定位精度为100%，召回率为95.8%，以IoU=0.5时mAP值为99.2%，IoU=0.5:0.95范围内的mAP值为69.1%。混合OCR系统在Auto-DG数据集上实现了89%的精度、65%的召回率和75%的F1分数，显著优于多个主流独立引擎的表现，证明其对于科学图像分析的可靠性。 大型语言模型不仅作为推理引擎工作，还充当智能助手提出后续步骤并验证结果。这种方法通过LLM代理的自动化手段，在SEM图像中显著提高了标尺检测与提取的效率和准确性，为微观结构分析提供了一个有价值的工具，并推动了科学成像领域的发展。
URL
https://arxiv.org/abs/2510.11260
