Abstract
Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.
Abstract (translated)
在混合现实（MR）中与真实世界对象进行交互常常因为这些对象过于拥挤、距离较远或部分被遮挡而变得困难，这阻碍了直接的选择和操作。我们观察到这些问题的根源在于直接在物理对象上执行交互时输入与其物理约束紧密耦合。我们的关键见解是通过引入代理——即现实世界对象的抽象表示来解除这种约束，从而使交互更加灵活。 Reality Proxy 是一个实现这一概念的系统，它可以在选择过程中将互动目标从实际物体无缝转移到它们对应的代理（proxies）。除了简化基本的选择操作之外，Reality Proxy 还使用人工智能技术为这些代理添加与相应物理对象相关的语义属性和空间层次关系。这使得在 MR 中进行诸如浏览、基于属性筛选、导航嵌套组以及复杂多对象选择等新型且以前难以实现的交互成为可能，并且无需引入新的手势或菜单系统。 我们展示了 Reality Proxy 在各种场景中的灵活性，包括办公信息检索、大规模空间导航和多无人机控制。专家评估表明该系统的实用性和易用性，并暗示基于代理（proxies）的抽象为未来的 MR 系统提供了一种强大而通用化的交互方式。
URL
https://arxiv.org/abs/2507.17248
