Abstract
Optical character recognition (OCR) and document understanding systems increasingly rely on large vision and vision-language models, yet evaluation remains centered on modern, Western, and institutional documents. This emphasis masks system behavior in historical and marginalized archives, where layout, typography, and material degradation shape interpretation. This study examines how OCR and document understanding systems are evaluated, with particular attention to Black historical newspapers. We review OCR and document understanding papers, as well as benchmark datasets, which are published between 2006 and 2025 using the PRISMA framework. We look into how the studies report training data, benchmark design, and evaluation metrics for vision transformer and multimodal OCR systems. During the review, we found that Black newspapers and other community-produced historical documents rarely appear in reported training data or evaluation benchmarks. Most evaluations emphasize character accuracy and task success on modern layouts. They rarely capture structural failures common in historical newspapers, including column collapse, typographic errors, and hallucinated text. To put these findings into perspective, we use previous empirical studies and archival statistics from significant Black press collections to show how evaluation gaps lead to structural invisibility and representational harm. We propose that these gaps occur due to organizational (meso) and institutional (macro) behaviors and structure, shaped by benchmark incentives and data governance decisions.
Abstract (translated)
光学字符识别（OCR）与文档理解系统日益依赖大规模视觉及视觉-语言模型，但现有评估仍集中于当代、西方及机构文档。这种侧重掩盖了系统在历史档案与边缘化文献中的表现——在这些场景中，版式、字体与材质退化会显著影响解读。本研究聚焦OCR与文档理解系统的评估方式，特别关注黑人历史报纸。我们采用PRISMA框架，系统回顾了2006至2025年间发表的OCR与文档理解论文及基准数据集，分析其中关于视觉Transformer与多模态OCR系统的训练数据、基准设计及评估指标报告方式。综述发现，黑人报纸及其他社群制作的历史文档极少出现在已报道的训练数据或评估基准中。多数评估强调现代版式下的字符准确率与任务成功率，却很少捕捉历史报纸常见的结构性问题，如栏目塌陷、字体错误与幻觉文本。为阐释这些发现，我们结合既往实证研究与重要黑人报刊收藏的档案统计，揭示评估差距如何导致结构性隐形与表征性伤害。我们认为此类差距源于组织（中观）与制度（宏观）层面的行为与结构，其形成受基准激励与数据治理决策共同塑造。
URL
https://arxiv.org/abs/2603.25761
