Abstract
Large language models (LLM) in natural language processing (NLP) have demonstrated great potential for in-context learning (ICL) -- the ability to leverage a few sets of example prompts to adapt to various tasks without having to explicitly update the model weights. ICL has recently been explored for computer vision tasks with promising early outcomes. These approaches involve specialized training and/or additional data that complicate the process and limit its generalizability. In this work, we show that off-the-shelf Stable Diffusion models can be repurposed for visual in-context learning (V-ICL). Specifically, we formulate an in-place attention re-computation within the self-attention layers of the Stable Diffusion architecture that explicitly incorporates context between the query and example prompts. Without any additional fine-tuning, we show that this repurposed Stable Diffusion model is able to adapt to six different tasks: foreground segmentation, single object detection, semantic segmentation, keypoint detection, edge detection, and colorization. For example, the proposed approach improves the mean intersection over union (mIoU) for the foreground segmentation task on Pascal-5i dataset by 8.9% and 3.2% over recent methods such as Visual Prompting and IMProv, respectively. Additionally, we show that the proposed method is able to effectively leverage multiple prompts through ensembling to infer the task better and further improve the performance.
Abstract (translated)
在自然语言处理（NLP）领域，大型语言模型（LLM）已经展示了在上下文学习（ICL）中的巨大潜力——即利用少量示例提示来适应各种任务的能力，而无需显式更新模型权重。最近有研究将ICL应用于计算机视觉任务，并取得了初步的积极成果。然而，这些方法通常需要专门的训练和/或额外的数据支持，这使得整个过程复杂且限制了其普适性。 在这项工作中，我们展示了现成的Stable Diffusion模型可以被重新部署用于视觉上下文学习（V-ICL）。具体来说，我们在Stable Diffusion架构的自注意力层中提出了一种就地注意力重构方法，该方法明确地将查询与示例提示之间的背景信息纳入考虑。无需任何额外微调的情况下，我们证明了这种重新部署后的Stable Diffusion模型可以适应六种不同的任务：前景分割、单对象检测、语义分割、关键点检测、边缘检测和着色。 例如，在Pascal-5i数据集的前景分割任务上，我们的方法提出的方案相对于最近的方法如Visual Prompting和IMProv分别提高了8.9%和3.2%的平均交并比（mIoU）。此外，我们还展示了所提出的方法能够通过集成多个提示来更有效地推断任务，并进一步提高性能。
URL
https://arxiv.org/abs/2508.09949
