Abstract
Synthesizing realistic human-object interactions (HOI) in video is challenging due to the complex, instance-specific interaction dynamics of both humans and objects. Incorporating controllability in video generation further adds to the complexity. Existing controllable video generation approaches face a trade-off: sparse controls like keypoint trajectories are easy to specify but lack instance-awareness, while dense signals such as optical flow, depths or 3D meshes are informative but costly to obtain. We propose VHOI, a two-stage framework that first densifies sparse trajectories into HOI mask sequences, and then fine-tunes a video diffusion model conditioned on these dense masks. We introduce a novel HOI-aware motion representation that uses color encodings to distinguish not only human and object motion, but also body-part-specific dynamics. This design incorporates a human prior into the conditioning signal and strengthens the model's ability to understand and generate realistic HOI dynamics. Experiments demonstrate state-of-the-art results in controllable HOI video generation. VHOI is not limited to interaction-only scenarios and can also generate full human navigation leading up to object interactions in an end-to-end manner. Project page: this https URL.
Abstract (translated)
在视频中合成逼真的人与物体交互（HOI）是具有挑战性的，因为人类和物体之间复杂的、特定实例的交互动态。将可控性加入到视频生成过程中进一步增加了复杂度。现有的可控制视频生成方法面临一个权衡：稀疏控制信号如关键点轨迹易于指定但缺乏具体实例的认识；而密集信号（例如光流、深度或3D网格）虽然信息量大，却获取成本高。 为此，我们提出了VHOI，这是一种两阶段框架，首先将稀疏的轨迹转化为HOI掩码序列，然后根据这些密集掩码对视频扩散模型进行微调。我们引入了一种新颖的人体交互感知运动表示方法，这种方法使用颜色编码来区分不仅人类和物体的移动性，还包含身体部位特定的动力学特征。这种设计把人的先验知识融入到了条件信号中，并增强了模型理解与生成逼真的HOI动态的能力。 实验表明，在可控人机互动视频生成方面VHOI达到了最先进的成果。值得注意的是，VHOI不仅仅局限于交互场景，还能以端到端的方式生成完整的从人体导航直至物体交互的全过程。项目的详细信息可以在这里找到：[项目页面](https://this/url)（请将URL替换为实际链接）。
URL
https://arxiv.org/abs/2512.09646
