Abstract
In this paper, we propose a solution for improving the quality of captions generated for figures in papers. We adopt the approach of summarizing the textual content in the paper to generate image captions. Throughout our study, we encounter discrepancies in the OCR information provided in the official dataset. To rectify this, we employ the PaddleOCR toolkit to extract OCR information from all images. Moreover, we observe that certain textual content in the official paper pertains to images that are not relevant for captioning, thereby introducing noise during caption generation. To mitigate this issue, we leverage LLaMA to extract image-specific information by querying the textual content based on image mentions, effectively filtering out extraneous information. Additionally, we recognize a discrepancy between the primary use of maximum likelihood estimation during text generation and the evaluation metrics such as ROUGE employed to assess the quality of generated captions. To bridge this gap, we integrate the BRIO model framework, enabling a more coherent alignment between the generation and evaluation processes. Our approach ranked first in the final test with a score of 4.49.
Abstract (translated)
在本文中，我们提出了提高论文中图表的摘要质量的解决方案。我们采用概述论文文本内容的策略来生成图像摘要。在整个研究过程中，我们遇到了官方数据集中提供OCR信息的差异。为了纠正这个问题，我们使用PaddleOCR工具包从所有图像中提取OCR信息。此外，我们还观察到官方论文中某些文本内容与标题无关，从而在生成标题时引入噪声。为了减轻这个问题，我们利用LLaMA提取图像特定信息，通过基于图像提及的文本内容进行查询，有效地过滤出无关信息。此外，我们认识到在文本生成中使用最大似然估计与评估指标如ROUGE之间存在差异。为了弥合这个差距，我们将BRIO模型框架集成到我们的方法中，实现了生成和评估过程之间的更一致性。我们的方法在最终测试中排名第一，得分4.49。
URL
https://arxiv.org/abs/2403.17342
