Abstract
With the continuous development of OCR technology and the expansion of application fields, text recognition in complex scenes has become a key challenge. Factors such as multiple fonts, mixed scenes and complex layouts seriously affect the recognition accuracy of traditional OCR models. Although OCR models based on deep learning have performed well in specific fields or similar data sets in recent years, the generalization ability and robustness of the model are still a big challenge when facing complex environments with multiple scenes. Furthermore, training an OCR model from scratch or fine-tuning all parameters is very demanding on computing resources and inference time, which limits the flexibility of its application. This study focuses on a fundamental aspect of mixed text recognition in response to the challenges mentioned above, which involves effectively fine-tuning the pre-trained basic OCR model to demonstrate exceptional performance across various downstream tasks. To this end, we propose a parameter-efficient hybrid text recognition method based on pre-trained OCR Transformer, namely DLoRA-TrOCR. This method embeds DoRA into the image encoder and LoRA into the internal structure of the text decoder, enabling efficient parameter fine-tuning for downstream tasks. Experimental results show that compared to similar parameter adjustment methods, our model DLoRA-TrOCR has the smallest number of parameters and performs better. It can achieve state-of-the-art performance on complex scene data sets involving simultaneous recognition of mixed handwritten, printed and street view texts.
Abstract (translated)
随着OCR技术的持续发展和应用领域的扩展，复杂场景中的文本识别已成为一个关键挑战。诸如多种字体、混合场景和复杂布局等因素，都严重影响了传统OCR模型的识别准确性。虽然基于深度学习的OCR模型在某些领域或类似数据集上的表现已经很好，但在面对复杂环境（多场景）时，模型的泛化能力和鲁棒性仍然是一个巨大的挑战。此外，从零开始训练OCR模型或微调所有参数在计算资源和推理时间上非常耗时，这限制了其应用的灵活性。本文关注于应对上述挑战的基本文本识别方面，这涉及通过预训练的基本OCR模型有效地微调以在各种下游任务上展示卓越性能。为此，我们提出了一个参数高效的混合文本识别方法，基于预训练的OCR Transformer，即DLoRA-TrOCR。该方法将DoRA嵌入到图像编码器中，将LoRA嵌入到文本解码器的内部结构中，以实现对下游任务的参数高效微调。实验结果表明，与类似的参数调整方法相比，我们的模型DLoRA-TrOCR具有最小的参数数量并表现出更好的性能。它可以在涉及同时识别混合手写、打印和街头街景文本的复杂场景数据集上实现最先进的性能。
URL
https://arxiv.org/abs/2404.12734
