Style_Transfer
-
ClimateGS: Real-Time Climate Simulation with 3D Gaussian Style Transfer
2025-03-19 03:01:35Yuezhen Xie, Meiying Zhang, Qi HaoAbstract
Adverse climate conditions pose significant challenges for autonomous systems, demanding reliable perception and decision-making across diverse environments. To better simulate these conditions, physically-based NeRF rendering methods have been explored for their ability to generate realistic scene representations. However, these methods suffer from slow rendering speeds and long preprocessing times, making them impractical for real-time testing and user interaction. This paper presents ClimateGS, a novel framework integrating 3D Gaussian representations with physical simulation to enable real-time climate effects rendering. The novelty of this work is threefold: 1) developing a linear transformation for 3D Gaussian photorealistic style transfer, enabling direct modification of spherical harmonics across bands for efficient and consistent style adaptation; 2) developing a joint training strategy for 3D style transfer, combining supervised and self-supervised learning to accelerate convergence while preserving original scene details; 3) developing a real-time rendering method for climate simulation, integrating physics-based effects with 3D Gaussian to achieve efficient and realistic rendering. We evaluate ClimateGS on MipNeRF360 and Tanks and Temples, demonstrating real-time rendering with comparable or superior visual quality to SOTA 2D/3D methods, making it suitable for interactive applications.
Abstract (translated)
恶劣的气候条件对自主系统构成了重大挑战，需要在各种环境中实现可靠感知和决策。为了更好地模拟这些条件，物理基础的NeRF渲染方法因其能够生成逼真的场景表示而受到探索。然而，这些方法存在渲染速度慢、预处理时间长的问题，这使得它们不适合实时测试和用户交互。本文介绍了ClimateGS，这是一个新的框架，它将3D高斯表示与物理仿真相结合，以实现气候效应的实时渲染。这项工作的创新性体现在三个方面： 1. **开发了用于3D高斯真实感风格迁移的线性变换**：通过直接修改各频带内的球谐函数（Spherical Harmonics），使高效的、一致的样式适应成为可能。 2. **提出了一种结合监督学习和自监督学习的联合训练策略，以加速收敛同时保持原始场景细节**。这种方法应用于3D风格迁移中。 3. **开发了一种用于气候模拟的实时渲染方法**，将基于物理的效果与3D高斯相结合，实现高效且逼真的渲染效果。 我们已经在MipNeRF360和Tanks and Temples数据集上评估了ClimateGS，并展示了其实时渲染能力，在视觉质量方面可与最新的2D/3D方法相媲美或超越。这些特性使其非常适合于互动应用。
URL
https://arxiv.org/abs/2503.14845
PDF
-
Snapmoji: Instant Generation of Animatable Dual-Stylized Avatars
2025-03-15 03:16:52Eric M. Chen, Di Liu, Sizhuo Ma, Michael Vasilkovsky, Bing Zhou, Qiang Gao, Wenzhou Wang, Jiahao Luo, Dimitris N. Metaxas, Vincent Sitzmann, Jian WangAbstract
The increasing popularity of personalized avatar systems, such as Snapchat Bitmojis and Apple Memojis, highlights the growing demand for digital self-representation. Despite their widespread use, existing avatar platforms face significant limitations, including restricted expressivity due to predefined assets, tedious customization processes, or inefficient rendering requirements. Addressing these shortcomings, we introduce Snapmoji, an avatar generation system that instantly creates animatable, dual-stylized avatars from a selfie. We propose Gaussian Domain Adaptation (GDA), which is pre-trained on large-scale Gaussian models using 3D data from sources such as Objaverse and fine-tuned with 2D style transfer tasks, endowing it with a rich 3D prior. This enables Snapmoji to transform a selfie into a primary stylized avatar, like the Bitmoji style, and apply a secondary style, such as Plastic Toy or Alien, all while preserving the user's identity and the primary style's integrity. Our system is capable of producing 3D Gaussian avatars that support dynamic animation, including accurate facial expression transfer. Designed for efficiency, Snapmoji achieves selfie-to-avatar conversion in just 0.9 seconds and supports real-time interactions on mobile devices at 30 to 40 frames per second. Extensive testing confirms that Snapmoji outperforms existing methods in versatility and speed, making it a convenient tool for automatic avatar creation in various styles.
Abstract (translated)
个性化头像系统（如 Snapchat Bitmojis 和 Apple Memojis）日益流行，这表明了数字自我表达需求的增长。尽管这些平台被广泛使用，但现有的头像生成工具仍然存在一些显著的局限性，包括由于预定义资产导致的表情限制、繁琐的自定义过程或低效的渲染要求。为了克服这些问题，我们提出了 Snapmoji，这是一种能够从用户自拍照片即时创建可动画化且具有双风格化的头像系统。 Snapmoji 使用高斯领域适应（Gaussian Domain Adaptation, GDA）技术进行工作，该技术预先在大型高斯模型上训练，并使用如 Objaverse 等来源的 3D 数据集进行了微调，使其具备丰富的 3D 先验知识。这使得 Snapmoji 能够将自拍照片转换为像 Bitmoji 这样的基础风格化头像，并可以应用额外的样式，例如塑料玩具或外星人风格，同时保持用户的身份和原始风格的真实性和完整性。 Snapmoji 系统能够生成支持动态动画的 3D 高斯头像，包括准确的表情传递。为了提高效率，Snapmoji 实现了在仅需 0.9 秒内完成自拍到头像转换，并且能够在移动设备上以每秒 30 到 40 帧的速度支持实时互动。广泛的测试表明，与现有方法相比，Snapmoji 在多样性和速度方面具有优势，使其成为创建各种风格的自动头像的理想工具。
URL
https://arxiv.org/abs/2503.11978
PDF
-
UStyle: Waterbody Style Transfer of Underwater Scenes by Depth-Guided Feature Synthesis
2025-03-14 21:49:40Md Abu Bakr Siddique, Junliang Liu, Piyush Singh, Md Jahidul IslamAbstract
The concept of waterbody style transfer remains largely unexplored in the underwater imaging and vision literature. Traditional image style transfer (STx) methods primarily focus on artistic and photorealistic blending, often failing to preserve object and scene geometry in images captured in high-scattering mediums such as underwater. The wavelength-dependent nonlinear attenuation and depth-dependent backscattering artifacts further complicate learning underwater image STx from unpaired data. This paper introduces UStyle, the first data-driven learning framework for transferring waterbody styles across underwater images without requiring prior reference images or scene information. We propose a novel depth-aware whitening and coloring transform (DA-WCT) mechanism that integrates physics-based waterbody synthesis to ensure perceptually consistent stylization while preserving scene structure. To enhance style transfer quality, we incorporate carefully designed loss functions that guide UStyle to maintain colorfulness, lightness, structural integrity, and frequency-domain characteristics, as well as high-level content in VGG and CLIP (contrastive language-image pretraining) feature spaces. By addressing domain-specific challenges, UStyle provides a robust framework for no-reference underwater image STx, surpassing state-of-the-art (SOTA) methods that rely solely on end-to-end reconstruction loss. Furthermore, we introduce the UF7D dataset, a curated collection of high-resolution underwater images spanning seven distinct waterbody styles, establishing a benchmark to support future research in underwater image STx. The UStyle inference pipeline and UF7D dataset are released at: this https URL.
Abstract (translated)
水体风格转换的概念在水下成像和视觉文献中仍然未被充分探索。传统图像风格转移（STx）方法主要集中在艺术性和逼真的融合上，往往无法保留高散射介质如水下环境中捕捉到的物体和场景几何结构。波长依赖性非线性衰减以及深度相关的背向散射伪影进一步复杂化了从未配对数据中学习水下图像风格转移的过程。本文介绍了UStyle，这是首个无需先验参考图像或场景信息即可实现跨水下图像间水体风格转换的数据驱动式学习框架。我们提出了一种新颖的基于深度感知的白化和着色变换（DA-WCT）机制，并将其与基于物理原理的水体合成相结合，以确保在保持场景结构的同时进行感知一致的风格化处理。为了提升风格转移的质量，我们设计了一系列精心挑选的损失函数，指导UStyle保留色彩丰富度、亮度、结构性质及频域特性，并且在VGG和CLIP（对比语言-图像预训练）特征空间中维持高层次的内容信息。通过解决特定领域的挑战，UStyle为无参考水下图像风格转移提供了一个稳健的框架，超越了仅依赖端到端重建损失的当前最先进技术。此外，我们还引入了UF7D数据集，这是一个精心挑选、涵盖七种不同水体样式的高分辨率水下图片集合，它建立了基准以支持未来在水下图像风格转换领域的研究。UStyle推理管线和UF7D数据集可在以下网址获取：此链接。
URL
https://arxiv.org/abs/2503.11893
PDF
-
PSF-4D: A Progressive Sampling Framework for View Consistent 4D Editing
2025-03-14 03:16:42Hasan Iqbal, Nazmul Karim, Umar Khalid, Azib Farooq, Zichun Zhong, Jing Hua, Chen ChenAbstract
Instruction-guided generative models, especially those using text-to-image (T2I) and text-to-video (T2V) diffusion frameworks, have advanced the field of content editing in recent years. To extend these capabilities to 4D scene, we introduce a progressive sampling framework for 4D editing (PSF-4D) that ensures temporal and multi-view consistency by intuitively controlling the noise initialization during forward diffusion. For temporal coherence, we design a correlated Gaussian noise structure that links frames over time, allowing each frame to depend meaningfully on prior frames. Additionally, to ensure spatial consistency across views, we implement a cross-view noise model, which uses shared and independent noise components to balance commonalities and distinct details among different views. To further enhance spatial coherence, PSF-4D incorporates view-consistent iterative refinement, embedding view-aware information into the denoising process to ensure aligned edits across frames and views. Our approach enables high-quality 4D editing without relying on external models, addressing key challenges in previous methods. Through extensive evaluation on multiple benchmarks and multiple editing aspects (e.g., style transfer, multi-attribute editing, object removal, local editing, etc.), we show the effectiveness of our proposed method. Experimental results demonstrate that our proposed method outperforms state-of-the-art 4D editing methods in diverse benchmarks.
Abstract (translated)
指令引导生成模型，尤其是使用文本到图像（T2I）和文本到视频（T2V）扩散框架的那些，在近年来推动了内容编辑领域的发展。为了将这些能力扩展到4D场景中，我们引入了一种渐进式采样框架——用于4D编辑的PSF-4D（Progressive Sampling Framework for 4D），通过直观地控制前向扩散过程中的噪声初始化来确保时间一致性和多视角一致性。 对于时间连贯性，我们设计了一个相关高斯噪声结构，该结构将帧按时间顺序链接起来，使得每一帧都能有意义地依赖于先前的帧。为了确保不同视角间的空间一致性，我们实现了一个跨视角噪声模型，它使用共享和独立的噪声成分来平衡各视角之间的共同点与独特细节。为增强空间连贯性，PSF-4D还集成了视图一致性的迭代细化过程，在去噪过程中嵌入了视图感知信息，以确保编辑在不同帧和视角之间的一致性和对齐。 我们的方法能够在不依赖外部模型的情况下实现高质量的4D编辑，解决了以前方法中的关键挑战。通过在多个基准测试和多种编辑方面（如风格转移、多属性编辑、物体移除、局部编辑等）进行广泛评估，我们展示了所提出方法的有效性。实验结果表明，在各种基准测试中，我们的方法优于现有的最先进的4D编辑技术。
URL
https://arxiv.org/abs/2503.11044
PDF
-
ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfer
2025-03-13 17:55:58Bolin Chen, Baoquan Zhao, Haoran Xie, Yi Cai, Qing Li, Xudong MaoAbstract
Style transfer involves transferring the style from a reference image to the content of a target image. Recent advancements in LoRA-based (Low-Rank Adaptation) methods have shown promise in effectively capturing the style of a single image. However, these approaches still face significant challenges such as content inconsistency, style misalignment, and content leakage. In this paper, we comprehensively analyze the limitations of the standard diffusion parameterization, which learns to predict noise, in the context of style transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method that enhances both content and style consistency by optimizing the LoRA weights to predict the original image rather than noise. We also propose a two-step training strategy that decouples the learning of content and style from the reference image. To effectively capture both the global structure and local details of the content image, we introduce a stepwise loss transition strategy. Additionally, we present an inference guidance method that enables continuous control over content and style strengths during inference. Through both qualitative and quantitative evaluations, our method demonstrates significant improvements in content and style consistency while effectively reducing content leakage.
Abstract (translated)
样式转换涉及将参考图像的风格转移到目标图像的内容上。最近，基于LoRA（低秩适应）的方法在捕捉单张图片的风格方面显示出巨大的潜力。然而，这些方法仍然面临诸如内容不一致、风格错位和内容泄漏等重大挑战。在这篇论文中，我们全面分析了标准扩散参数化（学习预测噪声）在样式转换中的局限性。为了解决这些问题，我们提出了ConsisLoRA，这是一种基于LoRA的方法，通过优化LoRA权重以预测原图而非噪声来增强内容和风格的一致性。此外，我们还提出了一种两步训练策略，将参考图像的内容和风格的学习过程解耦。为了有效地捕捉内容图像的全局结构和局部细节，我们引入了逐步损失转换策略。另外，我们还提出了一种推理指导方法，在推理过程中实现了对内容和风格强度的连续控制。通过定性和定量评估，我们的方法在内容和样式一致性方面表现出显著改进，并有效减少了内容泄漏。
URL
https://arxiv.org/abs/2503.10614
PDF
-
RoCo-Sim: Enhancing Roadside Collaborative Perception through Foreground Simulation
2025-03-13 14:33:42Yuwen Du, Anning Hu, Zichen Chao, Yifan Lu, Junhao Ge, Genjia Liu, Weitao Wu, Lanjun Wang, Siheng ChenAbstract
Roadside Collaborative Perception refers to a system where multiple roadside units collaborate to pool their perceptual data, assisting vehicles in enhancing their environmental awareness. Existing roadside perception methods concentrate on model design but overlook data issues like calibration errors, sparse information, and multi-view consistency, leading to poor performance on recent published datasets. To significantly enhance roadside collaborative perception and address critical data issues, we present the first simulation framework RoCo-Sim for road-side collaborative perception. RoCo-Sim is capable of generating diverse, multi-view consistent simulated roadside data through dynamic foreground editing and full-scene style transfer of a single image. RoCo-Sim consists of four components: (1) Camera Extrinsic Optimization ensures accurate 3D to 2D projection for roadside cameras; (2) A novel Multi-View Occlusion-Aware Sampler (MOAS) determines the placement of diverse digital assets within 3D space; (3) DepthSAM innovatively models foreground-background relationships from single-frame fixed-view images, ensuring multi-view consistency of foreground; and (4) Scalable Post-Processing Toolkit generates more realistic and enriched scenes through style transfer and other enhancements. RoCo-Sim significantly improves roadside 3D object detection, outperforming SOTA methods by 83.74 on Rcooper-Intersection and 83.12 on TUMTraf-V2X for AP70. RoCo-Sim fills a critical gap in roadside perception simulation. Code and pre-trained models will be released soon: this https URL
Abstract (translated)
路边协作感知指的是一个系统，其中多个路边单元（RSU）合作共享其感知数据，协助车辆提高对周围环境的感知能力。现有的路边感知方法主要集中在模型设计上，但忽视了校准误差、信息稀疏性和多视角一致性等数据问题，导致在最近发布的数据集上的表现不佳。为了显著增强路边协作感知并解决关键的数据问题，我们提出了首个用于道路边协同感知的仿真框架RoCo-Sim（Roadside Collaborative Simulation）。该框架能够通过动态前景编辑和单张图像的全场景风格转换生成多视角一致性的多样化模拟路边数据。 RoCo-Sim包含四个组成部分： 1. **Camera Extrinsic Optimization**：确保路边摄像头从3D到2D投影的准确性； 2. **Multi-View Occlusion-Aware Sampler (MOAS)**：一种新颖的采样器，用于在三维空间内放置各种数字资产，并考虑到多视角下的遮挡情况； 3. **DepthSAM**：创新性地建模单帧固定视角图像中的前景和背景关系，确保多视角下前景的一致性； 4. **Scalable Post-Processing Toolkit**：通过风格转换和其他增强手段生成更逼真且丰富化的场景。 RoCo-Sim显著提高了路边三维物体检测的性能，在Rcooper-Intersection数据集上超过现有最佳方法（SOTA）83.74%，在TUMTraf-V2X数据集上的AP70指标得分高于SOTA 83.12%。RoCo-Sim填补了路边感知仿真领域的关键空白，相关代码和预训练模型即将发布：[此链接](this https URL)（原文中的链接地址需要替换为实际的有效URL）。
URL
https://arxiv.org/abs/2503.10410
PDF
-
MoEdit: On Learning Quantity Perception for Multi-object Image Editing
2025-03-13 07:13:54Yanfeng Li, Kahou Chan, Yue Sun, Chantong Lam, Tong Tong, Zitong Yu, Keren Fu, Xiaohong Liu, Tao TanAbstract
Multi-object images are prevalent in various real-world scenarios, including augmented reality, advertisement design, and medical imaging. Efficient and precise editing of these images is critical for these applications. With the advent of Stable Diffusion (SD), high-quality image generation and editing have entered a new era. However, existing methods often struggle to consider each object both individually and part of the whole image editing, both of which are crucial for ensuring consistent quantity perception, resulting in suboptimal perceptual performance. To address these challenges, we propose MoEdit, an auxiliary-free multi-object image editing framework. MoEdit facilitates high-quality multi-object image editing in terms of style transfer, object reinvention, and background regeneration, while ensuring consistent quantity perception between inputs and outputs, even with a large number of objects. To achieve this, we introduce the Feature Compensation (FeCom) module, which ensures the distinction and separability of each object attribute by minimizing the in-between interlacing. Additionally, we present the Quantity Attention (QTTN) module, which perceives and preserves quantity consistency by effective control in editing, without relying on auxiliary tools. By leveraging the SD model, MoEdit enables customized preservation and modification of specific concepts in inputs with high quality. Experimental results demonstrate that our MoEdit achieves State-Of-The-Art (SOTA) performance in multi-object image editing. Data and codes will be available at this https URL.
Abstract (translated)
多对象图像在增强现实、广告设计和医学成像等多种实际场景中普遍存在。高效且精准地编辑这些图像对于上述应用至关重要。随着Stable Diffusion（SD）的出现，高质量的图像生成与编辑进入了一个新的时代。然而，现有的方法往往难以同时考虑每个对象的独立性和其作为整体图像的一部分的重要性，而这两种因素对保证一致的数量感知都至关重要，从而导致了次优的感知性能。 为了解决这些问题，我们提出了MoEdit——一种无辅助工具支持的多对象图像编辑框架。MoEdit使高质量的多对象图像编辑成为可能，包括风格转移、对象重新发明和背景再生，并确保输入与输出之间一致的数量感知，即使是在处理大量对象的情况下也是如此。为了实现这一点，我们引入了特征补偿（FeCom）模块，通过最小化中间交叉来保证每个对象属性的区别性和可分离性。此外，我们还提出了数量注意力（QTTN）模块，在编辑过程中有效控制以感知并保持数量的一致性，无需依赖辅助工具。 借助SD模型，MoEdit能够高质量地定制保存和修改输入中的特定概念。实验结果表明，我们的MoEdit在多对象图像编辑方面达到了最先进的性能水平。数据和代码将在以下链接提供：[此URL]（请将括号内的内容替换为实际提供的网址）。
URL
https://arxiv.org/abs/2503.10112
PDF
-
Fourier Decomposition for Explicit Representation of 3D Point Cloud Attributes
2025-03-13 05:13:40Donghyun Kim, Hyunah Ko, Chanyoung Kim, Seong Jae HwangAbstract
While 3D point clouds are widely utilized across various vision applications, their irregular and sparse nature make them challenging to handle. In response, numerous encoding approaches have been proposed to capture the rich semantic information of point clouds. Yet, a critical limitation persists: a lack of consideration for colored point clouds which are more capable 3D representations as they contain diverse attributes: color and geometry. While existing methods handle these attributes separately on a per-point basis, this leads to a limited receptive field and restricted ability to capture relationships across multiple points. To address this, we pioneer a point cloud encoding methodology that leverages 3D Fourier decomposition to disentangle color and geometric features while extending the receptive field through spectral-domain operations. Our analysis confirms that this encoding approach effectively separates feature components, where the amplitude uniquely captures color attributes and the phase encodes geometric structure, thereby enabling independent learning and utilization of both attributes. Furthermore, the spectral-domain properties of these components naturally aggregate local features while considering multiple points' information. We validate our point cloud encoding approach on point cloud classification and style transfer tasks, achieving state-of-the-art results on the DensePoint dataset with improvements via a proposed amplitude-based data augmentation strategy.
Abstract (translated)
尽管3D点云在各种视觉应用中被广泛使用，但它们的不规则性和稀疏性使其难以处理。为了捕捉点云丰富的语义信息，已经提出了许多编码方法。然而，一个关键限制仍然存在：缺乏对彩色点云的考虑，而彩色点云作为更强大的3D表示形式，包含颜色和几何属性等多种属性。现有的方法在每个点的基础上分别处理这些属性，这导致了有限的感受野以及捕获多个点之间关系的能力受限。 为了解决这个问题，我们首创了一种基于3D傅里叶分解的点云编码方法，这种方法能够分离颜色和几何特征，并通过频域操作扩展感受野。我们的分析表明，这种编码方式有效地分离了特征组件：幅度独特地捕捉颜色属性，而相位则编码了几何结构，从而实现了对两种属性的独立学习和使用。 此外，这些成分在频域上的特性自然聚合局部特征并考虑多个点的信息。我们在点云分类和风格转换任务上验证了我们的点云编码方法，并通过提出的一种基于幅度的数据增强策略，在DensePoint数据集上取得了最先进的结果。
URL
https://arxiv.org/abs/2503.10055
PDF
-
InteractEdit: Zero-Shot Editing of Human-Object Interactions in Images
2025-03-12 07:40:45Jiun Tian Hoe, Weipeng Hu, Wei Zhou, Chao Xie, Ziwei Wang, Chee Seng Chan, Xudong Jiang, Yap-Peng TanAbstract
This paper presents InteractEdit, a novel framework for zero-shot Human-Object Interaction (HOI) editing, addressing the challenging task of transforming an existing interaction in an image into a new, desired interaction while preserving the identities of the subject and object. Unlike simpler image editing scenarios such as attribute manipulation, object replacement or style transfer, HOI editing involves complex spatial, contextual, and relational dependencies inherent in humans-objects interactions. Existing methods often overfit to the source image structure, limiting their ability to adapt to the substantial structural modifications demanded by new interactions. To address this, InteractEdit decomposes each scene into subject, object, and background components, then employs Low-Rank Adaptation (LoRA) and selective fine-tuning to preserve pretrained interaction priors while learning the visual identity of the source image. This regularization strategy effectively balances interaction edits with identity consistency. We further introduce IEBench, the most comprehensive benchmark for HOI editing, which evaluates both interaction editing and identity preservation. Our extensive experiments show that InteractEdit significantly outperforms existing methods, establishing a strong baseline for future HOI editing research and unlocking new possibilities for creative and practical applications. Code will be released upon publication.
Abstract (translated)
本文介绍了InteractEdit，这是一种新颖的零样本人类-对象交互（HOI）编辑框架，旨在解决将图像中现有的交互转换为新的、期望的交互的任务，同时保持主体和客体的身份不变。与属性操控、物体替换或风格迁移等更简单的图像编辑场景不同，HOI 编辑涉及人类与物体之间复杂的空间、上下文和关系依赖性。现有方法往往过度适应源图像结构，限制了它们对新交互所需的重大结构修改的适应能力。为解决这一问题，InteractEdit 将每个场景分解为主体、客体和背景组件，并采用低秩适应（LoRA）和选择性微调来在保留预训练的交互先验的同时学习源图像的视觉身份。这种正则化策略有效地平衡了交互编辑与身份一致性。 我们进一步引入了IEBench，这是迄今为止最全面的HOI编辑基准测试工具，它评估了交互编辑和身份保持两个方面。我们的大量实验表明，InteractEdit 显著优于现有方法，在未来 HOI 编辑研究中建立了坚实的基础，并为创意和实用应用打开了新的可能性。论文发布后将公开代码。
URL
https://arxiv.org/abs/2503.09130
PDF
-
FPGS: Feed-Forward Semantic-aware Photorealistic Style Transfer of Large-Scale Gaussian Splatting
2025-03-11 23:52:56GeonU Kim, Kim Youwang, Lee Hyoseok, Tae-Hyun OhAbstract
We present FPGS, a feed-forward photorealistic style transfer method of large-scale radiance fields represented by Gaussian Splatting. FPGS, stylizes large-scale 3D scenes with arbitrary, multiple style reference images without additional optimization while preserving multi-view consistency and real-time rendering speed of 3D Gaussians. Prior arts required tedious per-style optimization or time-consuming per-scene training stage and were limited to small-scale 3D scenes. FPGS efficiently stylizes large-scale 3D scenes by introducing a style-decomposed 3D feature field, which inherits AdaIN's feed-forward stylization machinery, supporting arbitrary style reference images. Furthermore, FPGS supports multi-reference stylization with the semantic correspondence matching and local AdaIN, which adds diverse user control for 3D scene styles. FPGS also preserves multi-view consistency by applying semantic matching and style transfer processes directly onto queried features in 3D space. In experiments, we demonstrate that FPGS achieves favorable photorealistic quality scene stylization for large-scale static and dynamic 3D scenes with diverse reference images. Project page: this https URL
Abstract (translated)
我们介绍了一种名为FPGS的前馈式逼真风格转移方法，该方法用于通过高斯点表示的大规模辐射场。FPGS能够利用任意多个风格参考图像对大规模三维场景进行风格化处理，而无需额外优化，并且在保持多视角一致性和实时渲染速度的同时，保留了3D高斯分布的特性。相比之下，先前的技术需要繁琐的每种风格单独优化或耗时的每个场景训练阶段，并且仅限于小规模的三维场景。 FPGS通过引入一种风格分解的三维特征场，高效地对大规模三维场景进行样式化处理，这种方法继承了AdaIN（可适应实例归一化）的前馈式风格化机制，支持任意风格参考图像。此外，FPGS还通过语义匹配和局部AdaIN实现了多参考风格化，为三维场景提供了多样化的用户控制选项。 为了保持多视角一致性，FPGS直接在查询到的3D空间中的特征上应用语义匹配和样式传输过程。实验表明，FPGS能够对大规模静态和动态三维场景使用各种参考图像实现高质量、逼真的场景风格化处理。项目页面: [此链接](this https URL)
URL
https://arxiv.org/abs/2503.09635
PDF
-
YuE: Scaling Open Foundation Models for Long-Form Music Generation
2025-03-11 17:26:50Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, Xinrun Du, Zhen Ye, Tianyu Zheng, Yinghao Ma, Minghao Liu, Zeyue Tian, Ziya Zhou, Liumeng Xue, Xingwei Qu, Yizhi Li, Shangda Wu, Tianhao Shen, Ziyang Ma, Jun Zhan, Chunhui Wang, Yatian Wang, Xiaowei Chi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Shansong Liu, Lingrui Mei, Peng Li, Junjie Wang, Jianwei Yu, Guojian Pang, Xu Li, Zihao Wang, Xiaohuan Zhou, Lijun Yu, Emmanouil Benetos, Yong Chen, Chenghua Lin, Xie Chen, Gus Xia, Zhaoxiang Zhang, Chao Zhang, Wenhu Chen, Xinyu Zhou, Xipeng Qiu, Roger Dannenberg, Jiaheng Liu, Jian Yang, Wenhao Huang, Wei Xue, Xu Tan, Yike GuoAbstract
We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture. Specifically, YuE scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment. It achieves this through (1) track-decoupled next-token prediction to overcome dense mixture signals, (2) structural progressive conditioning for long-context lyrical alignment, and (3) a multitask, multiphase pre-training recipe to converge and generalize. In addition, we redesign the in-context learning technique for music generation, enabling versatile style transfer (e.g., converting Japanese city pop into an English rap while preserving the original accompaniment) and bidirectional generation. Through extensive evaluation, we demonstrate that YuE matches or even surpasses some of the proprietary systems in musicality and vocal agility. In addition, fine-tuning YuE enables additional controls and enhanced support for tail languages. Furthermore, beyond generation, we show that YuE's learned representations can perform well on music understanding tasks, where the results of YuE match or exceed state-of-the-art methods on the MARBLE benchmark. Keywords: lyrics2song, song generation, long-form, foundation model, music generation
Abstract (translated)
我们通过引入基于LLaMA2架构的开放基础模型YuE家族，解决了长篇音乐生成任务——尤其是具有挑战性的“歌词到歌曲”问题。具体来说，YuE可以处理万亿级的token，并且能够生成长达五分钟的音乐同时保持歌词的一致性、连贯的音乐结构和吸引人的声乐旋律以及适当的伴奏。它通过以下三种方式实现了这一目标： 1. 轨道解耦下一次令牌预测：克服密集混合信号。 2. 结构渐进条件设置：实现长时间上下文下的歌词一致性。 3. 多任务、多阶段预训练配方，以达成收敛和泛化。 此外，我们重新设计了音乐生成中的上下文学习技术，使其能够进行多样化的风格转换（例如，将日本城市流行乐转化为英文说唱，同时保留原有的伴奏）以及双向生成。通过广泛的评估，我们展示了YuE在音乐性和声乐敏捷性方面可以与一些专有系统匹敌甚至超越。此外，微调YuE还使额外的控制和对尾部语言的支持得到增强。 除了生成之外，我们还证明了YuE的学习表示还可以很好地应用于音乐理解任务，在MARBLE基准测试中，其结果可与现有最佳方法相匹配或超出。 关键词：歌词转歌曲、歌曲生成、长篇形式、基础模型、音乐生成
URL
https://arxiv.org/abs/2503.08638
PDF
-
CellStyle: Improved Zero-Shot Cell Segmentation via Style Transfer
2025-03-11 16:39:09R\"uveyda Yilmaz, Zhu Chen, Yuli Wu, Johannes StegmaierAbstract
Cell microscopy data are abundant; however, corresponding segmentation annotations remain scarce. Moreover, variations in cell types, imaging devices, and staining techniques introduce significant domain gaps between datasets. As a result, even large, pretrained segmentation models trained on diverse datasets (source datasets) struggle to generalize to unseen datasets (target datasets). To overcome this generalization problem, we propose CellStyle, which improves the segmentation quality of such models without requiring labels for the target dataset, thereby enabling zero-shot adaptation. CellStyle transfers the attributes of an unannotated target dataset, such as texture, color, and noise, to the annotated source dataset. This transfer is performed while preserving the cell shapes of the source images, ensuring that the existing source annotations can still be used while maintaining the visual characteristics of the target dataset. The styled synthetic images with the existing annotations enable the finetuning of a generalist segmentation model for application to the unannotated target data. We demonstrate that CellStyle significantly improves zero-shot cell segmentation performance across diverse datasets by finetuning multiple segmentation models on the style-transferred data. The code will be made publicly available.
Abstract (translated)
细胞显微镜数据非常丰富，但相应的分割标注却相对稀缺。此外，不同类型的细胞、成像设备以及染色技术之间的差异，在不同的数据集中引入了显著的领域差距。因此，即使是在多样化的数据集上经过大规模预训练的分割模型（源数据集），在面对未见过的数据集（目标数据集）时，仍然难以泛化。为了解决这一泛化问题，我们提出了一种名为CellStyle的方法，该方法可以在无需为目标数据集标注标签的情况下提高这些模型的分割质量，从而实现零样本适应。CellStyle 方法将未经注释的目标数据集中的属性（如纹理、颜色和噪声）转移到已注释的数据集中。在这一过程中，CellStyle 保持了源图像中细胞形状的完整性，确保现有的源标签仍然可以使用，并且保留目标数据集的视觉特征。这些带有现有标注的风格化合成图像使得通用分割模型可以在未标记的目标数据上进行微调。我们通过将多个分割模型在经过样式转换的数据上进行微调，证明了CellStyle 方法显著提升了跨多种数据集的零样本细胞分割性能。该代码将会公开发布。
URL
https://arxiv.org/abs/2503.08603
PDF
-
GAS-NeRF: Geometry-Aware Stylization of Dynamic Radiance Fields
2025-03-11 14:37:06Nhat Phuong Anh Vu, Abhishek Saroha, Or Litany, Daniel CremersAbstract
Current 3D stylization techniques primarily focus on static scenes, while our world is inherently dynamic, filled with moving objects and changing environments. Existing style transfer methods primarily target appearance -- such as color and texture transformation -- but often neglect the geometric characteristics of the style image, which are crucial for achieving a complete and coherent stylization effect. To overcome these shortcomings, we propose GAS-NeRF, a novel approach for joint appearance and geometry stylization in dynamic Radiance Fields. Our method leverages depth maps to extract and transfer geometric details into the radiance field, followed by appearance transfer. Experimental results on synthetic and real-world datasets demonstrate that our approach significantly enhances the stylization quality while maintaining temporal coherence in dynamic scenes.
Abstract (translated)
当前的3D风格化技术主要集中在静态场景上，而我们的世界本质上是动态的，充满了移动的对象和不断变化的环境。现有的风格迁移方法主要关注外观的转换，如颜色和纹理的变化，但往往忽视了样式图像的几何特征，这对于实现完整且连贯的风格化效果至关重要。为了克服这些不足，我们提出了GAS-NeRF，这是一种新颖的方法，用于动态辐射场中的联合外观和几何风格化。我们的方法利用深度图来提取并传输几何细节到辐射场中，随后进行外观转换。在合成数据集和真实世界数据集上的实验结果表明，我们的方法显著提高了风格化的质量，并且在动态场景中保持了时间一致性。
URL
https://arxiv.org/abs/2503.08483
PDF
-
DyArtbank: Diverse Artistic Style Transfer via Pre-trained Stable Diffusion and Dynamic Style Prompt Artbank
2025-03-11 12:56:47Zhanjie Zhang, Quanwei Zhang, Guangyuan Li, Junsheng Luan, Mengyuan Yang, Yun Wang, Lei ZhaoAbstract
Artistic style transfer aims to transfer the learned style onto an arbitrary content image. However, most existing style transfer methods can only render consistent artistic stylized images, making it difficult for users to get enough stylized images to enjoy. To solve this issue, we propose a novel artistic style transfer framework called DyArtbank, which can generate diverse and highly realistic artistic stylized images. Specifically, we introduce a Dynamic Style Prompt ArtBank (DSPA), a set of learnable parameters. It can learn and store the style information from the collection of artworks, dynamically guiding pre-trained stable diffusion to generate diverse and highly realistic artistic stylized images. DSPA can also generate random artistic image samples with the learned style information, providing a new idea for data augmentation. Besides, a Key Content Feature Prompt (KCFP) module is proposed to provide sufficient content prompts for pre-trained stable diffusion to preserve the detailed structure of the input content image. Extensive qualitative and quantitative experiments verify the effectiveness of our proposed method. Code is available: this https URL
Abstract (translated)
艺术风格转换旨在将学习到的风格转移到任意的内容图像上。然而，大多数现有的风格转移方法只能生成一致的艺术化效果图片，这使得用户难以获得足够多的美化后的图片以供享受。为了解决这个问题，我们提出了一种新的艺术风格转移框架——DyArtbank，它可以生成多样且高度逼真的艺术风格化图像。 具体来说，我们引入了一个动态风格提示艺术银行（Dynamic Style Prompt ArtBank, DSPA），这是一个可学习参数的集合，可以从艺术品收藏中学习并存储风格信息，并能够引导预训练的稳定扩散模型以生成多样化且高度真实的艺术风格化图像。此外，DSPA 还可以利用已学得的风格信息生成随机的艺术图片样本，为数据增强提供了新的思路。同时，我们还提出了一种关键内容特征提示（Key Content Feature Prompt, KCFP）模块来提供足够的内容提示给预训练的稳定扩散模型，以保留输入内容图像的详细结构。 大量的定性和定量实验验证了我们所提出方法的有效性。代码可从以下链接获取：[此 URL]
URL
https://arxiv.org/abs/2503.08392
PDF
-
U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers
2025-03-11 08:12:38Zhanjie Zhang, Ao Ma, Ke Cao, Jing Wang, Shanyuan Liu, Yuhang Ma, Bo Cheng, Dawei Leng, Yuhui YinAbstract
Ultra-high quality artistic style transfer refers to repainting an ultra-high quality content image using the style information learned from the style image. Existing artistic style transfer methods can be categorized into style reconstruction-based and content-style disentanglement-based style transfer approaches. Although these methods can generate some artistic stylized images, they still exhibit obvious artifacts and disharmonious patterns, which hinder their ability to produce ultra-high quality artistic stylized images. To address these issues, we propose a novel artistic image style transfer method, U-StyDiT, which is built on transformer-based diffusion (DiT) and learns content-style disentanglement, generating ultra-high quality artistic stylized images. Specifically, we first design a Multi-view Style Modulator (MSM) to learn style information from a style image from local and global perspectives, conditioning U-StyDiT to generate stylized images with the learned style information. Then, we introduce a StyDiT Block to learn content and style conditions simultaneously from a style image. Additionally, we propose an ultra-high quality artistic image dataset, Aes4M, comprising 10 categories, each containing 400,000 style images. This dataset effectively solves the problem that the existing style transfer methods cannot produce high-quality artistic stylized images due to the size of the dataset and the quality of the images in the dataset. Finally, the extensive qualitative and quantitative experiments validate that our U-StyDiT can create higher quality stylized images compared to state-of-the-art artistic style transfer methods. To our knowledge, our proposed method is the first to address the generation of ultra-high quality stylized images using transformer-based diffusion.
Abstract (translated)
超高质量的艺术风格迁移是指使用从风格图像中学习到的样式信息来重绘一张超高清晰度的内容图片。现有的艺术风格迁移方法可以分为基于风格重建和内容-风格分离的两种类型。尽管这些方法能够生成一些具有艺术风格化的图像，但它们仍然表现出明显的伪影和不和谐模式，这限制了其生成超高质量的艺术风格化图像的能力。为了解决这些问题，我们提出了一种新的艺术图像风格迁移方法U-StyDiT，该方法基于变压器扩散模型（DiT）构建，并通过学习内容-风格分离来生成超高质量的艺术风格化图像。 具体来说，我们首先设计了一个多视图样式调制器（MSM），用于从局部和全局视角学习一个风格图像中的样式信息，从而使U-StyDiT能够利用学到的样式信息生成艺术化的图像。然后，我们引入了StyDiT模块，该模块可以从一幅风格图片中同时学习到内容与样式的条件。 此外，我们还提出了一套用于超高质量的艺术图像数据集Aes4M，包含10个类别，每个类别中有40万张风格化图片。这个数据集有效地解决了现有的艺术风格迁移方法由于数据集大小和数据集中图片质量的原因而无法生成高品质艺术风格化图片的问题。 最后，广泛的定性和定量实验验证了我们的U-StyDiT能够创建出比当前最先进的艺术风格化图像更高的质量的图像。据我们所知，这是第一次使用基于变压器扩散模型的方法来生成超高质量的艺术风格化图像。
URL
https://arxiv.org/abs/2503.08157
PDF
-
Enhancing Retrieval for ESGLLM via ESG-CID -- A Disclosure Content Index Finetuning Dataset for Mapping GRI and ESRS
2025-03-10 18:07:33Shafiuddin Rehan Ahmed, Ankit Parag Shah, Quan Hung Tran, Vivek Khetan, Sukryool Kang, Ankit Mehta, Yujia Bao, Wei WeiAbstract
Climate change has intensified the need for transparency and accountability in organizational practices, making Environmental, Social, and Governance (ESG) reporting increasingly crucial. Frameworks like the Global Reporting Initiative (GRI) and the new European Sustainability Reporting Standards (ESRS) aim to standardize ESG reporting, yet generating comprehensive reports remains challenging due to the considerable length of ESG documents and variability in company reporting styles. To facilitate ESG report automation, Retrieval-Augmented Generation (RAG) systems can be employed, but their development is hindered by a lack of labeled data suitable for training retrieval models. In this paper, we leverage an underutilized source of weak supervision -- the disclosure content index found in past ESG reports -- to create a comprehensive dataset, ESG-CID, for both GRI and ESRS standards. By extracting mappings between specific disclosure requirements and corresponding report sections, and refining them using a Large Language Model as a judge, we generate a robust training and evaluation set. We benchmark popular embedding models on this dataset and show that fine-tuning BERT-based models can outperform commercial embeddings and leading public models, even under temporal data splits for cross-report style transfer from GRI to ESRS
Abstract (translated)
气候变化加剧了组织实践中透明度和问责制的需求，使得环境、社会和治理（ESG）报告变得愈加重要。全球报告倡议（Global Reporting Initiative, GRI）和新的欧洲可持续性报告标准（European Sustainability Reporting Standards, ESRS）等框架旨在标准化ESG报告的编制，然而由于ESG文件篇幅较长以及公司报告风格的变化多端，编写全面的ESG报告仍然具有挑战性。为了促进ESG报告自动化，可以采用检索增强生成（Retrieval-Augmented Generation, RAG）系统，但这类系统的开发因缺乏适合训练检索模型的标注数据而受限。 在本文中，我们利用了一个被忽视的弱监督来源——即过去ESG报告中的披露内容索引，创建了适用于GRI和ESRS标准的综合性数据集“ESG-CID”。通过提取特定披露要求与相应报告部分之间的映射，并使用大型语言模型进行校准，我们生成了一套稳健的训练及评估数据。我们在该数据集上对流行嵌入式模型进行了基准测试，并展示了通过对BERT基线模型微调，可以超越商业嵌入和领先公共模型的性能表现，即使是在跨报告风格转换（从GRI到ESRS）的时间分割数据下也是如此。 综上所述，本研究不仅贡献了一个全面的ESG披露索引数据集“ESG-CID”，还展示了通过大型语言模型指导下的微调过程可以有效提升检索增强生成系统的能力，在解决ESG自动化报告编制方面的挑战时展现出了巨大潜力。
URL
https://arxiv.org/abs/2503.10674
PDF
-
Balanced Image Stylization with Style Matching Score
2025-03-10 17:58:02Yuxin Jiang, Liming Jiang, Shuai Yang, Jia-Wei Liu, Ivor Tsang, Mike Zheng ShouAbstract
We present Style Matching Score (SMS), a novel optimization method for image stylization with diffusion models. Balancing effective style transfer with content preservation is a long-standing challenge. Unlike existing efforts, our method reframes image stylization as a style distribution matching problem. The target style distribution is estimated from off-the-shelf style-dependent LoRAs via carefully designed score functions. To preserve content information adaptively, we propose Progressive Spectrum Regularization, which operates in the frequency domain to guide stylization progressively from low-frequency layouts to high-frequency details. In addition, we devise a Semantic-Aware Gradient Refinement technique that leverages relevance maps derived from diffusion semantic priors to selectively stylize semantically important regions. The proposed optimization formulation extends stylization from pixel space to parameter space, readily applicable to lightweight feedforward generators for efficient one-step stylization. SMS effectively balances style alignment and content preservation, outperforming state-of-the-art approaches, verified by extensive experiments.
Abstract (translated)
我们提出了一种新颖的优化方法——风格匹配得分（SMS），用于基于扩散模型的图像风格化。在有效的风格转换与内容保留之间取得平衡是一个长期存在的挑战。不同于现有的努力，我们的方法将图像风格化重新定义为一个风格分布匹配的问题。目标风格分布是通过从现成的依赖于风格的LoRAs中经精心设计的评分函数来估计得出的。为了自适应地保持内容信息，我们提出了渐进频谱正则化技术，在频率域操作，逐步引导风格化过程，先从低频布局到高频细节进行处理。此外，我们还开发了一种语义感知梯度精炼技术，利用由扩散模型语义先验导出的相关图来选择性地对具有重要语义的区域进行风格化处理。所提出的优化方法将图像风格化的范围从像素空间扩展到了参数空间，并且能够轻松应用于轻量级前馈生成器以实现高效的一步式风格化过程。在广泛的实验验证下，SMS有效地平衡了风格一致性和内容保留，优于现有的最先进的方法。
URL
https://arxiv.org/abs/2503.07601
PDF
-
PersonaBooth: Personalized Text-to-Motion Generation
2025-03-10 14:38:00Boeun Kim, Hea In Jeong, JungHoon Sung, Yihua Cheng, Jeongmin Lee, Ju Yong Chang, Sang-Il Choi, Younggeun Choi, Saim Shin, Jungho Kim, Hyung Jin ChangAbstract
This paper introduces Motion Personalization, a new task that generates personalized motions aligned with text descriptions using several basic motions containing Persona. To support this novel task, we introduce a new large-scale motion dataset called PerMo (PersonaMotion), which captures the unique personas of multiple actors. We also propose a multi-modal finetuning method of a pretrained motion diffusion model called PersonaBooth. PersonaBooth addresses two main challenges: i) A significant distribution gap between the persona-focused PerMo dataset and the pretraining datasets, which lack persona-specific data, and ii) the difficulty of capturing a consistent persona from the motions vary in content (action type). To tackle the dataset distribution gap, we introduce a persona token to accept new persona features and perform multi-modal adaptation for both text and visuals during finetuning. To capture a consistent persona, we incorporate a contrastive learning technique to enhance intra-cohesion among samples with the same persona. Furthermore, we introduce a context-aware fusion mechanism to maximize the integration of persona cues from multiple input motions. PersonaBooth outperforms state-of-the-art motion style transfer methods, establishing a new benchmark for motion personalization.
Abstract (translated)
这篇论文介绍了Motion Personalization，这是一个新任务，旨在生成与文本描述相匹配的个性化动作序列，利用包含Persona的基本动作。为支持这一新颖的任务，我们引入了一个新的大规模动作数据集，称为PerMo（PersonaMotion），该数据集捕捉了多个演员的独特人格特征。此外，我们提出了一种基于预训练的动作扩散模型进行多模态微调的方法，名为PersonaBooth。 PersonaBooth解决了两个主要挑战：i) 专注于人物的PerMo数据集与缺乏特定人物信息的数据集之间的显著分布差距；ii) 从内容（动作类型）各异的动作中捕捉一致的人物特征的困难。为解决数据集分布差异问题，我们引入了一个接受新人物特性的角色令牌，并在微调期间对文本和视觉进行多模态适应。为了捕捉一致的人格特征，我们结合了对比学习技术，以增强具有相同人格特征样本之间的内部一致性。此外，我们还介绍了一种上下文感知融合机制，以最大化从多个输入动作中整合人物线索。 PersonaBooth在动作风格转移方法上超越了现有最佳水平，并为动作个性化建立了一个新的基准标准。
URL
https://arxiv.org/abs/2503.07390
PDF
-
Inversion-Free Video Style Transfer with Trajectory Reset Attention Control and Content-Style Bridging
Abstract
Video style transfer aims to alter the style of a video while preserving its content. Previous methods often struggle with content leakage and style misalignment, particularly when using image-driven approaches that aim to transfer precise styles. In this work, we introduce Trajectory Reset Attention Control (TRAC), a novel method that allows for high-quality style transfer while preserving content integrity. TRAC operates by resetting the denoising trajectory and enforcing attention control, thus enhancing content consistency while significantly reducing the computational costs against inversion-based methods. Additionally, a concept termed Style Medium is introduced to bridge the gap between content and style, enabling a more precise and harmonious transfer of stylistic elements. Building upon these concepts, we present a tuning-free framework that offers a stable, flexible, and efficient solution for both image and video style transfer. Experimental results demonstrate that our proposed framework accommodates a wide range of stylized outputs, from precise content preservation to the production of visually striking results with vibrant and expressive styles.
Abstract (translated)
视频风格转换的目标是改变视频的风格，同时保持其内容不变。以往的方法在处理内容渗漏和风格错配的问题上往往效果不佳，特别是在使用以图像驱动的方式进行精确风格转移时更为明显。在这项工作中，我们引入了一种名为轨迹重置注意力控制（Trajectory Reset Attention Control, TRAC）的新方法，该方法能够在保持内容完整性的前提下实现高质量的风格转换。TRAC通过重置去噪路径并强制执行注意力控制来运作，从而增强内容的一致性，并且显著降低了相对于基于逆向的方法的计算成本。 此外，我们还引入了一个名为“样式媒介”的概念，它连接了内容与风格之间的鸿沟，使更精确和谐地传输风格元素成为可能。基于这些概念，我们提出了一种无需调整的框架，提供了一种稳定、灵活且高效的解决方案，适用于图像和视频的风格转换。 实验结果表明，我们的框架能够支持多种多样化的风格化输出，从准确的内容保存到生产出视觉上引人注目的成果，具有生动而富有表现力的风格。
URL
https://arxiv.org/abs/2503.07363
PDF
-
AttenST: A Training-Free Attention-Driven Style Transfer Framework with Pre-Trained Diffusion Models
2025-03-10 13:28:36Bo Huang, Wenlun Xu, Qizhuo Han, Haodong Jing, Ying LiAbstract
While diffusion models have achieved remarkable progress in style transfer tasks, existing methods typically rely on fine-tuning or optimizing pre-trained models during inference, leading to high computational costs and challenges in balancing content preservation with style integration. To address these limitations, we introduce AttenST, a training-free attention-driven style transfer framework. Specifically, we propose a style-guided self-attention mechanism that conditions self-attention on the reference style by retaining the query of the content image while substituting its key and value with those from the style image, enabling effective style feature integration. To mitigate style information loss during inversion, we introduce a style-preserving inversion strategy that refines inversion accuracy through multiple resampling steps. Additionally, we propose a content-aware adaptive instance normalization, which integrates content statistics into the normalization process to optimize style fusion while mitigating the content degradation. Furthermore, we introduce a dual-feature cross-attention mechanism to fuse content and style features, ensuring a harmonious synthesis of structural fidelity and stylistic expression. Extensive experiments demonstrate that AttenST outperforms existing methods, achieving state-of-the-art performance in style transfer dataset.
Abstract (translated)
虽然扩散模型在风格转移任务中取得了显著进展，但现有方法通常依赖于微调或在推理过程中优化预训练模型，这导致了高昂的计算成本，并且在内容保留与风格融合之间难以取得平衡。为了解决这些局限性，我们引入了一种名为AttenST（Attention-Driven Style Transfer）的无需训练框架，它基于注意力驱动进行风格转移。 具体而言，我们提出了一种引导式自我注意机制，该机制通过保持内容图像的查询部分，而将关键值替换为样式图像中的对应部分来调整自我注意。这样可以有效地集成风格特征并避免额外的微调过程或优化步骤。为了减少逆向处理过程中样式的丢失，我们引入了一个样式保留逆向策略，通过多次重采样步骤提高逆向准确度。 此外，我们还提出了一种基于内容感知自适应实例归一化的方法，它将内容统计信息整合到归一化过程中以优化风格融合，并减轻对内容的破坏。为了确保结构精确性和风格表达之间的和谐结合，我们引入了双特征交叉注意力机制来融合内容和样式特征。 通过广泛的实验，证明AttenST优于现有方法，在风格转移数据集中实现了最先进的性能。
URL
https://arxiv.org/abs/2503.07307
PDF
