Abstract
Large pretrained diffusion models can provide strong priors beneficial for many graphics applications. However, generative applications such as neural rendering and inverse methods such as SVBRDF estimation and intrinsic image decomposition require additional input or output channels. Current solutions for channel expansion are often application specific and these solutions can be difficult to adapt to different diffusion models or new tasks. This paper introduces Teamwork: a flexible and efficient unified solution for jointly increasing the number of input and output channels as well as adapting a pretrained diffusion model to new tasks. Teamwork achieves channel expansion without altering the pretrained diffusion model architecture by coordinating and adapting multiple instances of the base diffusion model (\ie, teammates). We employ a novel variation of Low Rank-Adaptation (LoRA) to jointly address both adaptation and coordination between the different teammates. Furthermore Teamwork supports dynamic (de)activation of teammates. We demonstrate the flexibility and efficiency of Teamwork on a variety of generative and inverse graphics tasks such as inpainting, single image SVBRDF estimation, intrinsic decomposition, neural shading, and intrinsic image synthesis.
Abstract (translated)
大型预训练扩散模型可以为许多图形应用提供强有力的先验条件。然而，生成性应用程序（如神经渲染）和逆向方法（例如SVBRDF估计和固有图像分解）需要额外的输入或输出通道。当前针对通道扩展的解决方案通常是特定于具体应用场景的，并且这些方案很难适应不同的扩散模型或新的任务需求。本文介绍了“Teamwork”：一种灵活高效的统一解决方案，用于同时增加输入和输出通道的数量并使预训练的扩散模型能够适应新任务。“Teamwork”在不改变现有预训练扩散模型架构的情况下实现通道扩展，通过协调与调整多个基础扩散模型实例（即，“队友”）来完成这一目标。我们使用了一种新颖的低秩自适应（LoRA）变体，以同时解决不同“队友”之间的适配和协同问题。“Teamwork”还支持动态启用或停用这些“队友”。我们在多种生成性和逆向图形任务上展示了“Teamwork”的灵活性与效率，包括图像修复、单幅图像SVBRDF估计、固有分解、神经着色以及固有图像合成。
URL
https://arxiv.org/abs/2510.05532
