Abstract
Digital human video generation is gaining traction in fields like education and e-commerce, driven by advancements in head-body animation and lip-syncing technologies. However, realistic Hand-Object Interaction (HOI) - the complex dynamics between human hands and objects - continues to pose challenges. Generating natural and believable HOI reenactments is difficult due to issues such as occlusion between hands and objects, variations in object shapes and orientations, and the necessity for precise physical interactions, and importantly, the ability to generalize to unseen humans and objects. This paper presents a novel framework iDiT-HOI that enables in-the-wild HOI reenactment generation. Specifically, we propose a unified inpainting-based token process method, called Inp-TPU, with a two-stage video diffusion transformer (DiT) model. The first stage generates a key frame by inserting the designated object into the hand region, providing a reference for subsequent frames. The second stage ensures temporal coherence and fluidity in hand-object interactions. The key contribution of our method is to reuse the pretrained model's context perception capabilities without introducing additional parameters, enabling strong generalization to unseen objects and scenarios, and our proposed paradigm naturally supports long video generation. Comprehensive evaluations demonstrate that our approach outperforms existing methods, particularly in challenging real-world scenes, offering enhanced realism and more seamless hand-object interactions.
Abstract (translated)
数字人视频生成在教育和电子商务等领域正逐渐受到重视，这得益于头部与身体动画技术及唇同步技术的进步。然而，真实的“手-物交互”（HOI）——即人类双手与物体之间的复杂动态关系——仍然面临诸多挑战。由于手部与物体间的遮挡问题、不同形状和姿态的物体变化以及对精确物理互动的需求等原因，生成自然且可信的手物交互重演变得困难重重，尤其是当需要泛化到未见过的人体和物体时。 本文提出了一种名为iDiT-HOI的新框架，能够实现野外环境中的手物交互重现。具体来说，我们提出了一个统一的基于补全的标记处理方法——Inp-TPU，并采用了一个两阶段视频扩散变压器（DiT）模型。第一阶段通过在指定的手部区域插入目标物体来生成关键帧，为后续帧提供参考；第二阶段确保手物互动在时间上的连贯性和流畅性。 本方法的关键贡献在于，无需引入额外参数的情况下重用预训练模型的上下文感知能力，从而实现对未知对象和场景的强大泛化能力。此外，我们提出的范式自然支持长视频生成。全面评估表明，我们的方法优于现有技术，在具有挑战性的现实世界场景中表现出更强的真实感和更平滑的手物交互体验。
URL
https://arxiv.org/abs/2506.12847
