Abstract
We present InstructVTON, an instruction-following interactive virtual try-on system that allows fine-grained and complex styling control of the resulting generation, guided by natural language, on single or multiple garments. A computationally efficient and scalable formulation of virtual try-on formulates the problem as an image-guided or image-conditioned inpainting task. These inpainting-based virtual try-on models commonly use a binary mask to control the generation layout. Producing a mask that yields desirable result is difficult, requires background knowledge, might be model dependent, and in some cases impossible with the masking-based approach (e.g. trying on a long-sleeve shirt with "sleeves rolled up" styling on a person wearing long-sleeve shirt with sleeves down, where the mask will necessarily cover the entire sleeve). InstructVTON leverages Vision Language Models (VLMs) and image segmentation models for automated binary mask generation. These masks are generated based on user-provided images and free-text style instructions. InstructVTON simplifies the end-user experience by removing the necessity of a precisely drawn mask, and by automating execution of multiple rounds of image generation for try-on scenarios that cannot be achieved with masking-based virtual try-on models alone. We show that InstructVTON is interoperable with existing virtual try-on models to achieve state-of-the-art results with styling control.
Abstract (translated)
我们介绍了InstructVTON，这是一种遵循指令的交互式虚拟试穿系统，它允许通过自然语言引导，在单件或多件服装上实现细粒度和复杂的风格控制。该系统的虚拟试穿问题被表述为一个图像指导或条件下的修复任务（inpainting task），这是一个计算效率高且可扩展的方法。 基于修复技术的虚拟试穿模型通常使用二值掩码来控制生成布局。创建能够产生期望效果的掩模非常困难，需要特定知识，并可能依赖于具体模型；在某些情况下，仅靠掩码方法无法实现目标（例如，对于一个穿着长袖衬衫且袖子未卷起的人，想尝试长袖上衣并带有“袖子卷起来”的风格）。为了应对这些挑战，InstructVTON利用视觉语言模型(VLMs)和图像分割模型来自动生成二值掩模。这些掩模基于用户提供的图片和自由文本样式指令自动生成。 InstructVTON通过去除对精确绘制的掩码的需求以及自动化执行多轮图像生成的过程简化了最终用户的体验，尤其是在使用基于掩码的方法无法实现虚拟试穿的情况中更为明显。我们展示了InstructVTON可以与现有的虚拟试穿模型兼容，并在保持风格控制的同时达到最佳性能水平。 简而言之，InstructVTON通过结合视觉语言理解和自动化技术，显著提升了虚拟试穿系统的灵活性和用户友好性，同时实现了最先进的结果。
URL
https://arxiv.org/abs/2509.20524
