Abstract
Extended Reality (XR), including virtual, augmented, and mixed reality, provides immersive and interactive experiences across diverse applications, from VR-based education to AR-based assistance and MR-based training. However, widespread XR adoption remains limited due to two key challenges: 1) the high cost and complexity of authoring 3D content, especially for large-scale environments or complex interactions; and 2) the steep learning curve associated with non-intuitive interaction methods like handheld controllers or scripted gestures. Generative AI (GenAI) presents a promising solution by enabling intuitive, language-driven interaction and automating content generation. Leveraging vision-language models and diffusion-based generation, GenAI can interpret ambiguous instructions, understand physical scenes, and generate or manipulate 3D content, significantly lowering barriers to XR adoption. This paper explores the integration of XR and GenAI through three concrete use cases, showing how they address key obstacles in scalability and natural interaction, and identifying technical challenges that must be resolved to enable broader adoption.
Abstract (translated)
扩展现实（XR），包括虚拟现实（VR）、增强现实（AR）和混合现实（MR），为各种应用场景提供了沉浸式和互动体验，从基于 VR 的教育到基于 AR 的辅助功能以及基于 MR 的培训。然而，由于两个关键挑战，广泛采用 XR 仍然受到限制：1) 创作三维内容的高成本和复杂性，特别是对于大规模环境或复杂的交互；2) 手持控制器或脚本化手势等非直观互动方法的学习曲线陡峭。 生成式人工智能（GenAI）通过支持直觉化的语言驱动互动以及自动内容生成提供了一种有前景的解决方案。利用视觉-语言模型和扩散生成技术，GenAI 可以解读模糊指令、理解物理场景，并生成或操作三维内容，显著降低 XR 采用的门槛。本文探讨了通过三个具体用例将 XR 和 GenAI 集成的方法，展示了它们如何解决可扩展性和自然互动的关键障碍，并确定了必须解决的技术挑战，以便实现更广泛的采用。
URL
https://arxiv.org/abs/2601.15308
