Abstract
Cooking requires not only following instructions but also understanding, executing, and monitoring each step - a process that can be challenging without visual guidance. Although recipe images and videos offer helpful cues, they often lack consistency in focus, tools, and setup. To better support the cooking process, we introduce VisualChef, a method for generating contextual visual aids tailored to cooking scenarios. Given an initial frame and a specified action, VisualChef generates images depicting both the action's execution and the resulting appearance of the object, while preserving the initial frame's environment. Previous work aims to integrate knowledge extracted from large language models by generating detailed textual descriptions to guide image generation, which requires fine-grained visual-textual alignment and involves additional annotations. In contrast, VisualChef simplifies alignment through mask-based visual grounding. Our key insight is identifying action-relevant objects and classifying them to enable targeted modifications that reflect the intended action and outcome while maintaining a consistent environment. In addition, we propose an automated pipeline to extract high-quality initial, action, and final state frames. We evaluate VisualChef quantitatively and qualitatively on three egocentric video datasets and show its improvements over state-of-the-art methods.
Abstract (translated)
烹饪不仅需要遵循指示，还需要理解、执行和监控每一个步骤——这一过程在缺乏视觉指导的情况下可能会变得具有挑战性。虽然食谱图片和视频提供了有用的提示，但它们往往在焦点、工具和设置方面缺乏一致性。为了更好地支持烹饪过程，我们引入了VisualChef，这是一种生成针对烹饪场景的上下文可视化辅助的方法。给定初始画面和指定的动作，VisualChef可以生成描绘动作执行以及对象最终外观的图像，同时保留初始画面中的环境。 以往的工作旨在通过从大型语言模型中提取知识来生成详细的文本描述以指导图像生成，这要求精细的视觉-文本对齐，并涉及额外的标注。相比之下，VisualChef通过基于掩码的视觉定位简化了对齐过程。我们的关键洞察是识别与动作相关的对象并对其进行分类，以便进行针对性修改，反映预期的动作和结果的同时保持环境的一致性。 此外，我们提出了一种自动管道来提取高质量的初始、动作和最终状态帧。我们在三个第一人称视频数据集上从定量和定性的角度评估了VisualChef，并展示了它相比现有最先进方法的进步。
URL
https://arxiv.org/abs/2506.18569
