Abstract
The virtual content in augmented reality (AR) can introduce misleading or harmful information, leading to semantic misunderstandings or user errors. In this work, we focus on visual information manipulation (VIM) attacks in AR where virtual content changes the meaning of real-world scenes in subtle but impactful ways. We introduce a taxonomy that categorizes these attacks into three formats: character, phrase, and pattern manipulation, and three purposes: information replacement, information obfuscation, and extra wrong information. Based on the taxonomy, we construct a dataset, AR-VIM. It consists of 452 raw-AR video pairs spanning 202 different scenes, each simulating a real-world AR scenario. To detect such attacks, we propose a multimodal semantic reasoning framework, VIM-Sense. It combines the language and visual understanding capabilities of vision-language models (VLMs) with optical character recognition (OCR)-based textual analysis. VIM-Sense achieves an attack detection accuracy of 88.94% on AR-VIM, consistently outperforming vision-only and text-only baselines. The system reaches an average attack detection latency of 7.07 seconds in a simulated video processing framework and 7.17 seconds in a real-world evaluation conducted on a mobile Android AR application.
Abstract (translated)
在增强现实（AR）中，虚拟内容可能会引入误导性或有害的信息，导致语义误解或用户错误。在这项工作中，我们关注的是视觉信息操纵（VIM）攻击，在这种攻击中，虚拟内容以微妙但有影响力的方式改变了真实场景的意义。我们提出了一个分类法，将这些攻击分为三种格式：字符、短语和模式操作，以及三个目的：信息替换、信息混淆和额外错误信息。 基于这一分类法，我们构建了一个数据集AR-VIM，其中包括452对原始AR视频片段，涵盖了202种不同的场景，每一种都模拟了现实中的AR情景。为了检测此类攻击，我们提出了一种多模态语义推理框架VIM-Sense，该框架结合了视觉-语言模型（VLM）的语言和视觉理解能力以及基于光学字符识别（OCR）的文本分析。 在AR-VIM数据集上，VIM-Sense实现了88.94%的攻击检测准确率，并且始终优于仅使用视觉或仅使用文本的基础方法。该系统在一个模拟视频处理框架中达到平均7.07秒的攻击检测延迟，在移动Android AR应用程序的真实世界评估中达到了平均7.17秒的延迟。
URL
https://arxiv.org/abs/2507.20356
