Abstract
Video Virtual Try-On (VVT) aims to simulate the natural appearance of garments across consecutive video frames, capturing their dynamic variations and interactions with human body motion. However, current VVT methods still face challenges in terms of spatiotemporal consistency and garment content preservation. First, they use diffusion models based on the U-Net, which are limited in their expressive capability and struggle to reconstruct complex details. Second, they adopt a separative modeling approach for spatial and temporal attention, which hinders the effective capture of structural relationships and dynamic consistency across frames. Third, their expression of garment details remains insufficient, affecting the realism and stability of the overall synthesized results, especially during human motion. To address the above challenges, we propose MagicTryOn, a video virtual try-on framework built upon the large-scale video diffusion this http URL replace the U-Net architecture with a diffusion Transformer and combine full self-attention to jointly model the spatiotemporal consistency of videos. We design a coarse-to-fine garment preservation strategy. The coarse strategy integrates garment tokens during the embedding stage, while the fine strategy incorporates multiple garment-based conditions, such as semantics, textures, and contour lines during the denoising stage. Moreover, we introduce a mask-aware loss to further optimize garment region fidelity. Extensive experiments on both image and video try-on datasets demonstrate that our method outperforms existing SOTA methods in comprehensive evaluations and generalizes to in-the-wild scenarios.
Abstract (translated)
视频虚拟试穿（VVT）旨在模拟服装在连续视频帧中的自然外观，捕捉其动态变化及其与人体动作的互动。然而，当前的VVT方法仍然面临着时空一致性和衣物内容保留方面的挑战。 首先，它们使用基于U-Net的扩散模型，这种模型表达能力有限，在重建复杂细节时遇到困难。 其次，它们采用分离式建模方法来处理空间和时间注意力问题，这阻碍了对帧间结构关系及动态一致性有效捕捉的能力。 第三，他们对于衣物细节的表现仍然不足，影响整体合成结果的真实感与稳定性，尤其是在人体动作期间。 为了解决上述挑战，我们提出了MagicTryOn框架。该框架基于大规模视频扩散模型构建，并用扩散Transformer替代U-Net架构，同时结合全面的自注意力机制来共同建模视频的时空一致性。我们设计了一种从粗到细的衣物保留策略：粗略策略在嵌入阶段集成服装令牌，而精细策略则在去噪阶段整合多个基于衣物条件的因素（如语义、纹理和轮廓线）。此外，我们引入了一个遮罩感知损失来进一步优化衣物区域的真实感。 通过针对图像及视频试穿数据集的广泛实验表明，我们的方法在全面评估中超越了现有的最先进的技术，并且能够应用于现实场景。
URL
https://arxiv.org/abs/2505.21325
