Abstract
Modeling and synthesizing complex hand-object interactions remains a significant challenge, even for state-of-the-art physics engines. Conventional simulation-based approaches rely on explicitly defined rigid object models and pre-scripted hand gestures, making them inadequate for capturing dynamic interactions with non-rigid or articulated entities such as deformable fabrics, elastic materials, hinge-based structures, furry surfaces, or even living creatures. In this paper, we present SpriteHand, an autoregressive video generation framework for real-time synthesis of versatile hand-object interaction videos across a wide range of object types and motion patterns. SpriteHand takes as input a static object image and a video stream in which the hands are imagined to interact with the virtual object embedded in a real-world scene, and generates corresponding hand-object interaction effects in real time. Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence. Our 1.3B model supports real-time streaming generation at around 18 FPS and 640x368 resolution, with an approximate 150 ms latency on a single NVIDIA RTX 5090 GPU, and more than a minute of continuous output. Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.
Abstract (translated)
对给定文本的翻译如下： 复杂的手部与物体之间的交互建模和合成仍然是一个重大挑战，即使是当前最先进的物理引擎也无法完全解决。传统基于仿真的方法依赖于明确定义的刚性对象模型以及预设的手势脚本，这使得它们在捕捉非刚性或关节结构（如可变形织物、弹性材料、铰链结构、毛茸表面甚至活体生物）与手部之间的动态交互时显得力不从心。本文中，我们介绍了SpriteHand——这是一个用于实时生成多种类型物体和运动模式下的手部-物体互动视频的自回归视频生成框架。SpriteHand以一个静态的对象图像和一段假想的手部在现实场景中的虚拟对象上进行互动的视频流作为输入，并实时生成相应的真实感手部与对象之间的交互效果。我们的模型采用因果推理架构来进行自回归生成，并利用混合后训练方法来增强视觉真实性和时间一致性。我们的1.3B参数量的模型支持以约每秒18帧和640x368分辨率进行实时流式生成，在单个NVIDIA RTX 5090 GPU上的延迟约为150毫秒，且可以持续输出超过一分钟的内容。实验结果表明，我们的方法在视觉质量、物理合理性及交互真实度方面均优于基于生成和引擎的基线模型。
URL
https://arxiv.org/abs/2512.01960
