Abstract
Video virtual try-on aims to naturally fit a garment to a target person in consecutive video frames. It is a challenging task, on the one hand, the output video should be in good spatial-temporal consistency, on the other hand, the details of the given garment need to be preserved well in all the frames. Naively using image-based try-on methods frame by frame can get poor results due to severe inconsistency. Recent diffusion-based video try-on methods, though very few, happen to coincide with a similar solution: inserting temporal attention into image-based try-on model to adapt it for video try-on task, which have shown improvements but there still exist inconsistency problems. In this paper, we propose ViTI (Video Try-on Inpainter), formulate and implement video virtual try-on as a conditional video inpainting task, which is different from previous methods. In this way, we start with a video generation problem instead of an image-based try-on problem, which from the beginning has a better spatial-temporal consistency. Specifically, at first we build a video inpainting framework based on Diffusion Transformer with full 3D spatial-temporal attention, and then we progressively adapt it for video garment inpainting, with a collection of masking strategies and multi-stage training. After these steps, the model can inpaint the masked garment area with appropriate garment pixels according to the prompt with good spatial-temporal consistency. Finally, as other try-on methods, garment condition is added to the model to make sure the inpainted garment appearance and details are as expected. Both quantitative and qualitative experimental results show that ViTI is superior to previous works.
Abstract (translated)
视频虚拟试穿的目标是将衣物自然地合身到目标人物在连续帧中的姿态上。这项任务具有挑战性，一方面输出的视频应具备良好的时空一致性；另一方面，给定衣物的所有细节都需要在每一帧中得到良好保留。简单地逐帧应用基于图像的试穿方法会导致严重的不一致问题。尽管最近有一些扩散模型（diffusion-based）的方法尝试解决这个问题，但它们通过将时间注意力机制插入到现有的基于图像的试穿模型来适应视频任务，虽然取得了一些进步，但仍存在一致性的问题。 在本文中，我们提出了ViTI（Video Try-on Inpainter），并将其定义为一个条件下的视频修复任务。这种方法不同于以往的方法，因为它从一开始就以视频生成问题而非单一图像为基础，从而更好地保证了时空的一致性。具体来说，我们首先建立了一个基于扩散变换器（Diffusion Transformer）的3D空间时间注意力框架，并逐步适应该框架用于衣物区域的修复工作，通过一系列掩码策略和多阶段训练来实现这一点。经过这些步骤后，模型可以在遮挡的衣物区域内根据提示进行填充，生成具有良好时空一致性的适当衣物像素。 最后，如同其他试穿方法一样，在这一过程中加入服装条件以确保修复后的服装外观及细节符合预期要求。无论是定量还是定性实验结果都显示ViTI优于先前的工作。
URL
https://arxiv.org/abs/2506.21270
