Abstract
Creating physically realistic content in VR often requires complex modeling tools or predefined 3D models, textures, and animations, which present significant barriers for non-expert users. In this paper, we propose SketchPlay, a novel VR interaction framework that transforms humans' air-drawn sketches and gestures into dynamic, physically realistic scenes, making content creation intuitive and playful like drawing. Specifically, sketches capture the structure and spatial arrangement of objects and scenes, while gestures convey physical cues such as velocity, direction, and force that define movement and behavior. By combining these complementary forms of input, SketchPlay captures both the structure and dynamics of user-created content, enabling the generation of a wide range of complex physical phenomena, such as rigid body motion, elastic deformation, and cloth dynamics. Experimental results demonstrate that, compared to traditional text-driven methods, SketchPlay offers significant advantages in expressiveness, and user experience. By providing an intuitive and engaging creation process, SketchPlay lowers the entry barrier for non-expert users and shows strong potential for applications in education, art, and immersive storytelling.
Abstract (translated)
在虚拟现实中创建物理上逼真的内容通常需要复杂的建模工具或预定义的3D模型、纹理和动画，这些都为非专业用户设置了相当大的障碍。在这篇论文中，我们提出了SketchPlay，这是一种新颖的VR交互框架，它将人类在空气中的手绘草图和手势转化为动态且物理上逼真的场景，使内容创作直观而有趣，就像画画一样。具体而言，草图捕捉了物体和场景的结构和空间布局，而手势传达了诸如速度、方向和力等物理线索，这些定义了运动和行为。通过结合这两种互补形式的输入，SketchPlay 能够同时捕获用户创建内容的结构和动态性，从而生成广泛的复杂物理现象，包括刚体运动、弹性变形和布料动力学。 实验结果表明，与传统的基于文本的方法相比，SketchPlay 在表现力和用户体验方面具有显著优势。通过提供直观且引人入胜的内容创作过程，SketchPlay 降低了非专业用户的入门门槛，并显示出在教育、艺术和沉浸式叙事等领域的强大应用潜力。
URL
https://arxiv.org/abs/2512.22016
