Abstract
When obtaining visual illustrations from text descriptions, today's methods take a description with-a single text context caption, or an action description-and retrieve or generate the matching visual context. However, prior work does not permit visual illustration of multistep descriptions, e.g. a cooking recipe composed of multiple steps. Furthermore, simply handling each step description in isolation would result in an incoherent demonstration. We propose Stitch-a-Recipe, a novel retrieval-based method to assemble a video demonstration from a multistep description. The resulting video contains clips, possibly from different sources, that accurately reflect all the step descriptions, while being visually coherent. We formulate a training pipeline that creates large-scale weakly supervised data containing diverse and novel recipes and injects hard negatives that promote both correctness and coherence. Validated on in-the-wild instructional videos, Stitch-a-Recipe achieves state-of-the-art performance, with quantitative gains up to 24% as well as dramatic wins in a human preference study.
Abstract (translated)
在从文本描述获取视觉图解时，当今的方法通常采用单一的文本背景说明或动作描述来检索或生成相应的视觉内容。然而，现有的方法并不能处理多步骤描述的可视化展示，例如由多个步骤组成的烹饪食谱。此外，仅仅孤立地处理每个步骤的描述会导致不连贯的表现形式。为此，我们提出了Stitch-a-Recipe这一创新性的基于检索的方法，用于从多步描述中组装视频演示。该方法生成的视频包含来自不同来源的片段，并能够准确反映所有步骤描述的同时保持视觉上的连贯性。 为了训练这个模型，我们设计了一个训练流程，创建了大规模弱监督数据集，其中包含了多样且新颖的食谱，并引入难例负样本以促进正确性和连贯性的提升。在现实世界中的教学视频上进行验证后，Stitch-a-Recipe取得了最先进的性能，在定量评估中最多提高了24%，并且在人类偏好的研究中也有显著的优势。
URL
https://arxiv.org/abs/2503.13821
