Abstract
In this study, we formulate an OCR-free sequence generation model for visual document understanding (VDU). Our model not only parses text from document images but also extracts the spatial coordinates of the text based on the multi-head architecture. Named as Coordinate-aware End-to-end Document Parser (CREPE), our method uniquely integrates these capabilities by introducing a special token for OCR text, and token-triggered coordinate decoding. We also proposed a weakly-supervised framework for cost-efficient training, requiring only parsing annotations without high-cost coordinate annotations. Our experimental evaluations demonstrate CREPE's state-of-the-art performances on document parsing tasks. Beyond that, CREPE's adaptability is further highlighted by its successful usage in other document understanding tasks such as layout analysis, document visual question answering, and so one. CREPE's abilities including OCR and semantic parsing not only mitigate error propagation issues in existing OCR-dependent methods, it also significantly enhance the functionality of sequence generation models, ushering in a new era for document understanding studies.
Abstract (translated)
在这项研究中，我们提出了一个无需光学字符识别（OCR）的视觉文档理解（VDU）序列生成模型。我们的模型不仅解析了文档图像中的文本，还根据多头架构提取了文本的空间坐标。我们为其命名为“ Coordinate-aware End-to-end Document Parser (CREPE)”，通过引入一个特殊标记来标记OCR文本，并实现标记触发的位置解码。我们还提出了一种弱监督的训练框架，只需解析无高成本坐标注释的标注即可。我们的实验评估结果表明，CREPE在文档解析任务中具有最先进的性能。除此之外，CREPE在其他文档理解任务（如布局分析、文档视觉问答等）中的应用也进一步证明了其灵活性。CREPE的包括OCR和语义解析的能力不仅减轻了现有OCR依赖方法中的错误传播问题，还显著增强了序列生成模型的功能，引领了文档理解研究的新纪元。
URL
https://arxiv.org/abs/2405.00260
