Abstract
Scene generation is crucial to many computer graphics applications. Recent advances in generative AI have streamlined sketch-to-image workflows, easing the workload for artists and designers in creating scene concept art. However, these methods often struggle for complex scenes with multiple detailed objects, sometimes missing small or uncommon instances. In this paper, we propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation after reviewing the entire cross-attention mechanism. This scheme revitalizes the existing ControlNet model, enabling effective handling of multi-instance generations, involving prompt balance, characteristics prominence, and dense tuning. Specifically, this approach enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances. It also includes a characteristics prominence module that highlights TopK indices in each channel, ensuring essential features are better represented based on token sketches. Additionally, it employs dense tuning to refine contour details in the attention map, compensating for instance-related regions. Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models. It consistently generates detailed, multi-instance 2D images, closely adhering to the input prompts and enhancing visual quality in complex multi-instance scenes. Code is available at this https URL.
Abstract (translated)
场景生成对许多计算机图形应用至关重要。最近在生成式人工智能方面的进展简化了草图转图像的工作流程，减轻了艺术家和设计师创作场景概念艺术的工作量。然而，这些方法往往难以处理包含多个详细对象的复杂场景，有时会遗漏小或不常见的实例。在这篇论文中，我们在审查整个交叉注意力机制后提出了一个无需训练的三元组调整方案（T3-S2S）以进行草图转场景生成。该方案重新激活了现有的ControlNet模型，使其能够有效处理多实例生成，包括提示平衡、特征突出和密集调优。具体而言，这种方法通过提示平衡模块增强了关键词表示，降低了遗漏关键实例的风险。它还包含一个特征突出模块，在每个通道中突出TopK索引，确保基于标记草图的重要特征得到更好的表现。此外，该方法采用密集调优来细化注意力图中的轮廓细节，弥补与实例相关的区域。实验验证了我们的三元组调整方案显著提高了现有草图转图像模型的性能。它能够一致地生成详细、多实例的2D图像，紧密遵循输入提示并增强复杂多实例场景下的视觉质量。代码可在此 https URL 获取。
URL
https://arxiv.org/abs/2412.13486
