Abstract
Retrieval-Augmented Generation (RAG) systems depend critically on the quality of document preprocessing, yet no prior study has evaluated PDF processing frameworks by their impact on downstream question-answering accuracy. We address this gap through a systematic comparison of four open-source PDF-to-Markdown conversion frameworks, Docling, MinerU, Marker, and DeepSeek OCR, across 19 pipeline configurations for extracting text and other contents from PDFs, varying the conversion tool, cleaning transformations, splitting strategy, and metadata enrichment. Evaluation was performed using a manually curated 50-question benchmark over a corpus of 36 Portuguese administrative documents (1,706 pages, ~492K words), with LLM-as-judge scoring averaged over 10 runs. Two baselines bounded the results: naïve PDFLoader (86.9%) and manually curated Markdown (97.1%). Docling with hierarchical splitting and image descriptions achieved the highest automated accuracy (94.1%). Metadata enrichment and hierarchy-aware chunking contributed more to accuracy than the conversion framework choice alone. Font-based hierarchy rebuilding consistently outperformed LLM-based approaches. An exploratory GraphRAG implementation scored only 82%, underperforming basic RAG, suggesting that naïve knowledge graph construction without ontological guidance does not yet justify its added complexity. These findings demonstrate that data preparation quality is the dominant factor in RAG system performance.
Abstract (translated)
检索增强生成（RAG）系统的性能高度依赖于文档预处理的质量，然而此前尚无研究通过评估下游问答准确率来比较PDF处理框架。本研究通过系统对比四个开源PDF转Markdown转换框架（Docling、MinerU、Marker和DeepSeek OCR），在19种流水线配置下从PDF中提取文本及其他内容，涵盖转换工具、清洗转换、切分策略和元数据增强四个变量。评估基于手动构建的50道问题基准测试，针对36份葡萄牙行政文件（共1,706页，约49.2万字）的语料库进行，采用LLM作为评判者并取10次运行的平均分。两个基线限定了结果范围：朴素PDFLoader（86.9%）与手动整理的Markdown（97.1%）。采用分层切分和图像描述的Docling实现了最高的自动化准确率（94.1%）。元数据增强与分层感知的文本切分对准确率的提升贡献大于转换框架本身的选择。基于字体的层级重建方法持续优于基于LLM的方法。探索性GraphRAG实现仅得82分，表现低于基础RAG，表明缺乏本体论指导的朴素知识图谱构建尚未证明其额外复杂性的合理性。这些发现证明，数据准备质量是决定RAG系统性能的主导因素。
URL
https://arxiv.org/abs/2604.04948
