Abstract
Current sketch extraction methods either require extensive training or fail to capture a wide range of artistic styles, limiting their practical applicability and versatility. We introduce Mixture-of-Self-Attention (MixSA), a training-free sketch extraction method that leverages strong diffusion priors for enhanced sketch perception. At its core, MixSA employs a mixture-of-self-attention technique, which manipulates self-attention layers by substituting the keys and values with those from reference sketches. This allows for the seamless integration of brushstroke elements into initial outline images, offering precise control over texture density and enabling interpolation between styles to create novel, unseen styles. By aligning brushstroke styles with the texture and contours of colored images, particularly in late decoder layers handling local textures, MixSA addresses the common issue of color averaging by adjusting initial outlines. Evaluated with various perceptual metrics, MixSA demonstrates superior performance in sketch quality, flexibility, and applicability. This approach not only overcomes the limitations of existing methods but also empowers users to generate diverse, high-fidelity sketches that more accurately reflect a wide range of artistic expressions.
Abstract (translated)
目前的草图提取方法要么需要大量的训练，要么无法捕捉广泛的艺术风格，这限制了它们的实际应用性和灵活性。我们引入了一种无需训练的草图提取方法——混合自注意力（MixSA），该方法利用强大的扩散先验来增强对草图的理解和感知。在核心机制上，MixSA采用了混合自注意力技术，通过用参考草图中的键值替换自我注意层中的键值来进行操作。这使得能够将笔触元素无缝地整合到初始轮廓图像中，并提供对纹理密度的精确控制以及风格之间的插值以创造新颖、前所未见的风格。 通过使笔触样式与彩色图像的纹理和轮廓对齐，特别是在处理局部纹理的解码器后期层中进行调整，MixSA解决了常见的颜色平均化问题。经过各种感知度量评估后，结果表明MixSA在草图质量、灵活性和应用性方面表现优越。这种方法不仅克服了现有方法的局限性，还使用户能够生成多样化且高保真的草图，更准确地反映广泛的艺术表达形式。
URL
https://arxiv.org/abs/2501.00816
