Abstract
Visual Document Understanding (VDU) systems have achieved strong performance in information extraction by integrating textual, layout, and visual signals. However, their robustness under realistic adversarial perturbations remains insufficiently explored. We introduce the first unified framework for generating and evaluating multi-modal adversarial attacks on OCR-based VDU models. Our method covers six gradient-based layout attack scenarios, incorporating manipulations of OCR bounding boxes, pixels, and texts across both word and line granularities, with constraints on layout perturbation budget (e.g., IoU >= 0.6) to preserve plausibility. Experimental results across four datasets (FUNSD, CORD, SROIE, DocVQA) and six model families demonstrate that line-level attacks and compound perturbations (BBox + Pixel + Text) yield the most severe performance degradation. Projected Gradient Descent (PGD)-based BBox perturbations outperform random-shift baselines in all investigated models. Ablation studies further validate the impact of layout budget, text modification, and adversarial transferability.
Abstract (translated)
视觉文档理解（VDU）系统通过整合文本、布局和视觉信号，在信息提取方面取得了显著的性能。然而，这些系统的鲁棒性在面对现实中的对抗性扰动时仍不够充分研究。我们介绍了首个用于生成和评估基于OCR的VDU模型多模态对抗攻击的统一框架。我们的方法涵盖了六种基于梯度的布局攻击场景，包括对OCR边界框、像素和文本（从单词级别到行级别）的操作，并在布局干扰预算（例如IoU >= 0.6）上设置了约束条件以保持现实性。通过四个数据集（FUNSD、CORD、SROIE、DocVQA）及六种模型家族的实验结果表明，行级别的攻击和复合扰动（边界框+像素+文本）导致性能下降最为严重。基于投影梯度下降（PGD）的边界框扰动在所有研究的模型中均优于随机偏移基线。进一步的消融研究表明了布局预算、文本修改以及对抗迁移性的影响力。
URL
https://arxiv.org/abs/2506.16407
