Style_Transfer
-
TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation
2025-03-10 08:35:51Victor Shea-Jay Huang, Le Zhuo, Yi Xin, Zhaokai Wang, Peng Gao, Hongsheng LiAbstract
Diffusion Transformers (DiTs) are a powerful yet underexplored class of generative models compared to U-Net-based diffusion models. To bridge this gap, we introduce TIDE (Temporal-aware Sparse Autoencoders for Interpretable Diffusion transformErs), a novel framework that enhances temporal reconstruction within DiT activation layers across denoising steps. TIDE employs Sparse Autoencoders (SAEs) with a sparse bottleneck layer to extract interpretable and hierarchical features, revealing that diffusion models inherently learn hierarchical features at multiple levels (e.g., 3D, semantic, class) during generative pre-training. Our approach achieves state-of-the-art reconstruction performance, with a mean squared error (MSE) of 1e-3 and a cosine similarity of 0.97, demonstrating superior accuracy in capturing activation dynamics along the denoising trajectory. Beyond interpretability, we showcase TIDE's potential in downstream applications such as sparse activation-guided image editing and style transfer, enabling improved controllability for generative systems. By providing a comprehensive training and evaluation protocol tailored for DiTs, TIDE contributes to developing more interpretable, transparent, and trustworthy generative models.
Abstract (translated)
扩散变换器（DiT）相较于基于U-Net的扩散模型，是一个强大但尚未充分探索的生成模型类别。为了缩小这一差距，我们引入了TIDE（用于可解释性扩散转换器的时间感知稀疏自编码框架），这是一个新的框架，在扩散过程中增强了时间重建能力，并在整个去噪步骤中改善了DiT激活层的表现。 TIDE利用具有稀疏瓶颈层的稀疏自编码器（SAE）来提取可解释和分层特征，揭示了扩散模型在生成预训练期间内在地学习多层级特征（例如3D、语义、类别）。我们的方法实现了最先进的重建性能，在均方误差（MSE）为1e-3和余弦相似度为0.97的情况下表现出色，证明其在捕捉去噪轨迹上的激活动力学方面具有出色的准确性。 除了可解释性之外，我们展示了TIDE在下游应用中的潜力，如稀疏激活引导的图像编辑和风格转移，使生成系统获得了更好的可控性。通过提供专门针对DiTs的综合训练和评估协议，TIDE有助于开发更可解释、透明和值得信赖的生成模型。
URL
https://arxiv.org/abs/2503.07050
PDF
-
SOYO: A Tuning-Free Approach for Video Style Morphing via Style-Adaptive Interpolation in Diffusion Models
2025-03-10 07:27:01Haoyu Zheng, Qifan Yu, Binghe Yu, Yang Dai, Wenqiao Zhang, Juncheng Li, Siliang Tang, Yueting ZhuangAbstract
Diffusion models have achieved remarkable progress in image and video stylization. However, most existing methods focus on single-style transfer, while video stylization involving multiple styles necessitates seamless transitions between them. We refer to this smooth style transition between video frames as video style morphing. Current approaches often generate stylized video frames with discontinuous structures and abrupt style changes when handling such transitions. To address these limitations, we introduce SOYO, a novel diffusion-based framework for video style morphing. Our method employs a pre-trained text-to-image diffusion model without fine-tuning, combining attention injection and AdaIN to preserve structural consistency and enable smooth style transitions across video frames. Moreover, we notice that applying linear equidistant interpolation directly induces imbalanced style morphing. To harmonize across video frames, we propose a novel adaptive sampling scheduler operating between two style images. Extensive experiments demonstrate that SOYO outperforms existing methods in open-domain video style morphing, better preserving the structural coherence of video frames while achieving stable and smooth style transitions.
Abstract (translated)
扩散模型在图像和视频风格化方面取得了显著的进展。然而，大多数现有的方法主要集中在单一风格转换上，而涉及多风格的视频风格化则需要这些风格之间能够无缝过渡。我们称这种视频帧之间的平滑风格转变过程为“视频风格变形”。当前的方法往往在处理这类过渡时会产生结构不连续且风格变化突兀的问题。为了克服这些问题，我们引入了SOYO，这是一种用于视频风格变形的新颖扩散框架。我们的方法利用预先训练好的文本到图像的扩散模型而不进行微调，并结合注意力注入和AdaIN（归一化激活）技术以保持结构一致性并实现视频帧之间平滑的风格过渡。此外，我们注意到直接应用线性等距插值会导致不均衡的风格变形。为了在整个视频帧中保持一致，我们提出了一种新的自适应采样调度器，在两种样式图像之间运行。广泛的实验表明，SOYO在开放领域的视频风格变形方面优于现有方法，不仅更好地保留了视频帧的结构一致性，还实现了稳定且平滑的风格过渡。
URL
https://arxiv.org/abs/2503.06998
PDF
-
Dr Genre: Reinforcement Learning from Decoupled LLM Feedback for Generic Text Rewriting
2025-03-09 21:23:52Yufei Li, John Nham, Ganesh Jawahar, Lei Shu, David Uthus, Yun-Hsuan Sung, Chengrun Yang, Itai Rolnick, Yi Qiao, Cong LiuAbstract
Generic text rewriting is a prevalent large language model (LLM) application that covers diverse real-world tasks, such as style transfer, fact correction, and email editing. These tasks vary in rewriting objectives (e.g., factual consistency vs. semantic preservation), making it challenging to develop a unified model that excels across all dimensions. Existing methods often specialize in either a single task or a specific objective, limiting their generalizability. In this work, we introduce a generic model proficient in factuality, stylistic, and conversational rewriting tasks. To simulate real-world user rewrite requests, we construct a conversational rewrite dataset, ChatRewrite, that presents ``natural''-sounding instructions, from raw emails using LLMs. Combined with other popular rewrite datasets, including LongFact for the factuality rewrite task and RewriteLM for the stylistic rewrite task, this forms a broad benchmark for training and evaluating generic rewrite models. To align with task-specific objectives, we propose Dr Genre, a Decoupled-reward learning framework for Generic rewriting, that utilizes objective-oriented reward models with a task-specific weighting. Evaluation shows that \approach delivers higher-quality rewrites across all targeted tasks, improving objectives including instruction following (agreement), internal consistency (coherence), and minimal unnecessary edits (conciseness).
Abstract (translated)
通用文本重写是一项广泛应用于大型语言模型（LLM）的实际任务，涵盖了多种现实世界的应用场景，例如文体转换、事实校正以及电子邮件编辑。这些任务在重写目标方面各有不同（如：事实一致性 vs 语义保持），这使得开发一个能在所有维度上表现优秀的统一模型变得具有挑战性。现有的方法通常专注于单一任务或特定目标，从而限制了它们的通用性。在这项工作中，我们引入了一个擅长于事实性、风格性和对话式重写任务的通用模型。为了模拟现实世界中用户对于文本重写的请求，我们构建了一个基于对话式的重写数据集ChatRewrite，该数据集提供了使用LLM处理原始邮件时“自然”听起来的指令。结合LongFact（用于事实一致性重写任务）和RewriteLM（用于风格性重写任务）等其他流行的数据集，形成了一个广泛的基准测试环境以训练和评估通用文本重写模型。 为了使模型更符合特定的任务目标，我们提出了Dr Genre（Decoupled-reward learning framework for Generic rewriting），即解耦奖励学习框架。此框架使用面向具体目标的奖励模型，并结合了任务特异性的加权策略。评估结果表明，\approach 在所有被关注的任务中提供了更高质量的重写文本，提升了包括指令遵循（一致性）、内部连贯性（语义一致）和最小化不必要的修改（简洁度）在内的多个指标表现。
URL
https://arxiv.org/abs/2503.06781
PDF
-
PhysicsGen: Can Generative Models Learn from Images to Predict Complex Physical Relations?
2025-03-07 11:19:13Martin Spitznagel, Jan Vaillant, Janis KeuperAbstract
The image-to-image translation abilities of generative learning models have recently made significant progress in the estimation of complex (steered) mappings between image distributions. While appearance based tasks like image in-painting or style transfer have been studied at length, we propose to investigate the potential of generative models in the context of physical simulations. Providing a dataset of 300k image-pairs and baseline evaluations for three different physical simulation tasks, we propose a benchmark to investigate the following research questions: i) are generative models able to learn complex physical relations from input-output image pairs? ii) what speedups can be achieved by replacing differential equation based simulations? While baseline evaluations of different current models show the potential for high speedups (ii), these results also show strong limitations toward the physical correctness (i). This underlines the need for new methods to enforce physical correctness. Data, baseline models and evaluation code this http URL.
Abstract (translated)
最近，生成式学习模型在估计图像分布之间的复杂（偏转）映射方面取得了显著进展。虽然已经对基于外观的任务（如图像修复或风格转换）进行了深入研究，但我们提议探讨生成式模型在物理模拟中的潜力。为此，我们提供了一个包含30万张图像配对的数据集以及针对三种不同物理仿真任务的基准评估。我们的目标是通过以下研究问题来探究这一领域： i) 生成式模型是否能够从输入-输出图像对中学习复杂的物理关系？ ii) 替换基于微分方程的模拟可以实现多大的加速效果？ 虽然当前各种模型的基础性评估显示了实现高加速率（ii）的巨大潜力，但这些结果也表明在确保物理正确性方面存在严重局限（i）。这强调了需要开发新的方法来强制执行物理正确性。数据集、基准模型和评估代码可在此网址获取：[请在这里插入链接]。
URL
https://arxiv.org/abs/2503.05333
PDF
-
Iris Style Transfer: Enhancing Iris Recognition with Style Features and Privacy Preservation through Neural Style Transfer
2025-03-06 18:55:21Mengdi Wang, Efe Bozkir, Enkelejda KasneciAbstract
Iris texture is widely regarded as a gold standard biometric modality for authentication and identification. The demand for robust iris recognition methods, coupled with growing security and privacy concerns regarding iris attacks, has escalated recently. Inspired by neural style transfer, an advanced technique that leverages neural networks to separate content and style features, we hypothesize that iris texture's style features provide a reliable foundation for recognition and are more resilient to variations like rotation and perspective shifts than traditional approaches. Our experimental results support this hypothesis, showing a significantly higher classification accuracy compared to conventional features. Further, we propose using neural style transfer to mask identifiable iris style features, ensuring the protection of sensitive biometric information while maintaining the utility of eye images for tasks like eye segmentation and gaze estimation. This work opens new avenues for iris-oriented, secure, and privacy-aware biometric systems.
Abstract (translated)
虹膜纹理被广泛认为是身份认证和识别的黄金标准生物特征模态。随着对可靠虹膜识别方法的需求日益增加，以及关于虹膜攻击的安全性和隐私问题的关注加剧，最近人们对这一领域产生了更大的兴趣。受神经风格迁移（一种利用神经网络分离内容与风格特征的先进技术）启发，我们假设虹膜纹理中的风格特征可以为识别提供一个可靠的依据，并且比传统方法更能抵抗旋转和透视变化带来的影响。我们的实验结果支持了这一假设，显示出相比传统的特征提取方法，分类准确率有了显著提升。 此外，我们提出使用神经风格迁移来掩盖可识别人脸风格的虹膜特征，以确保敏感生物识别信息的安全性，同时保持眼睛图像在诸如眼球分割和注视估计等任务中的实用性。这项工作为面向虹膜、安全且隐私意识强的生物识别系统开辟了新的途径。
URL
https://arxiv.org/abs/2503.04707
PDF
-
ObjMST: An Object-Focused Multimodal Style Transfer Framework
2025-03-06 11:55:44Chanda Grover Kamra, Indra Deep Mastan, Debayan GuptaAbstract
We propose ObjMST, an object-focused multimodal style transfer framework that provides separate style supervision for salient objects and surrounding elements while addressing alignment issues in multimodal representation learning. Existing image-text multimodal style transfer methods face the following challenges: (1) generating non-aligned and inconsistent multimodal style representations; and (2) content mismatch, where identical style patterns are applied to both salient objects and their surrounding elements. Our approach mitigates these issues by: (1) introducing a Style-Specific Masked Directional CLIP Loss, which ensures consistent and aligned style representations for both salient objects and their surroundings; and (2) incorporating a salient-to-key mapping mechanism for stylizing salient objects, followed by image harmonization to seamlessly blend the stylized objects with their environment. We validate the effectiveness of ObjMST through experiments, using both quantitative metrics and qualitative visual evaluations of the stylized outputs. Our code is available at: this https URL.
Abstract (translated)
我们提出了ObjMST，这是一种以对象为中心的多模态风格转换框架，它为显著对象和周围元素提供了独立的风格监督，并解决了多模态表示学习中的对齐问题。现有的图像-文本多模态风格转换方法面临以下挑战：(1) 生成不一致且不对齐的多模态风格表示；(2) 内容不匹配，在这种情况下，相同的风格模式会被应用于显著对象及其周围元素上。 我们的方法通过以下方式缓解这些问题： (1) 引入了一种特定于风格的掩码方向CLIP损失（Style-Specific Masked Directional CLIP Loss），确保了对于显著对象及其周围环境的一致且对齐的风格表示； (2) 采用了一种从显著到关键映射机制，用于给显著对象进行风格化处理，之后通过图像调和来无缝地将这些经过风格化的对象与其背景融合在一起。 我们通过实验验证了ObjMST的有效性，其中包括定量指标以及对风格化输出的定性视觉评估。我们的代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2503.04353
PDF
-
SCSA: A Plug-and-Play Semantic Continuous-Sparse Attention for Arbitrary Semantic Style Transfer
2025-03-06 05:56:25Chunnan Shang, Zhizhong Wang, Hongwei Wang, Xiangming MengAbstract
Attention-based arbitrary style transfer methods, including CNN-based, Transformer-based, and Diffusion-based, have flourished and produced high-quality stylized images. However, they perform poorly on the content and style images with the same semantics, i.e., the style of the corresponding semantic region of the generated stylized image is inconsistent with that of the style image. We argue that the root cause lies in their failure to consider the relationship between local regions and semantic regions. To address this issue, we propose a plug-and-play semantic continuous-sparse attention, dubbed SCSA, for arbitrary semantic style transfer -- each query point considers certain key points in the corresponding semantic region. Specifically, semantic continuous attention ensures each query point fully attends to all the continuous key points in the same semantic region that reflect the overall style characteristics of that region; Semantic sparse attention allows each query point to focus on the most similar sparse key point in the same semantic region that exhibits the specific stylistic texture of that region. By combining the two modules, the resulting SCSA aligns the overall style of the corresponding semantic regions while transferring the vivid textures of these regions. Qualitative and quantitative results prove that SCSA enables attention-based arbitrary style transfer methods to produce high-quality semantic stylized images.
Abstract (translated)
基于注意力的任意风格转换方法，包括CNN（卷积神经网络）基础、Transformer基础和扩散模型基础的方法已经蓬勃发展，并生成了高质量的艺术化图像。然而，当处理内容图与样式图具有相同语义的情况时，这些方法表现不佳，即所生成的艺术化图像中对应语义区域的风格与其参考样式的风格不一致。我们认为根本原因在于它们未能考虑局部区域和语义区域之间的关系。为解决这一问题，我们提出了一种插件式连续稀疏注意力机制（SCSA），专门用于任意语义风格转换——每个查询点都会考虑到其对应语义区域内的特定关键点。 具体而言，连续的语义注意确保了每个查询点能够全面关注同语义区域内反映整体风格特征的所有连续关键点；而稀疏的语义注意则使每个查询点能聚焦于在相同语义区域中展示该区域独特风格纹理的最相似的那个单一关键点。通过结合这两个模块，SCSA能够在进行风格转换时同时对齐对应语义区域的整体风格并转移这些区域的具体纹理细节。 定性和定量结果证明了SCSA能使基于注意力机制的方法产生高质量的语义艺术化图像，并解决了当内容和样式图具有相同语义时生成的艺术化图像中语义区域间风格不一致的问题。
URL
https://arxiv.org/abs/2503.04119
PDF
-
Modality-Agnostic Style Transfer for Holistic Feature Imputation
2025-03-03 07:09:24Seunghun Baek, Jaeyoon Sim, Mustafa Dere, Minjeong Kim, Guorong Wu, Won Hwa KimAbstract
Characterizing a preclinical stage of Alzheimer's Disease (AD) via single imaging is difficult as its early symptoms are quite subtle. Therefore, many neuroimaging studies are curated with various imaging modalities, e.g., MRI and PET, however, it is often challenging to acquire all of them from all subjects and missing data become inevitable. In this regards, in this paper, we propose a framework that generates unobserved imaging measures for specific subjects using their existing measures, thereby reducing the need for additional examinations. Our framework transfers modality-specific style while preserving AD-specific content. This is done by domain adversarial training that preserves modality-agnostic but AD-specific information, while a generative adversarial network adds an indistinguishable modality-specific style. Our proposed framework is evaluated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) study and compared with other imputation methods in terms of generated data quality. Small average Cohen's $d$ $< 0.19$ between our generated measures and real ones suggests that the synthetic data are practically usable regardless of their modality type.
Abstract (translated)
通过单一影像手段来表征阿尔茨海默病（AD）的临床前期阶段是困难的，因为其早期症状非常细微。因此，许多神经影像学研究通常会结合使用多种成像模式（如MRI和PET），然而，从所有受试者中获取所有这些模态的数据往往具有挑战性，并且不可避免地会出现数据缺失的情况。为此，在本文中，我们提出了一种框架，该框架能够根据现有测量结果生成特定受试者的未观察到的影像指标，从而减少对额外检查的需求。我们的框架在转移特异性模式风格的同时保持了AD特有的内容信息。这通过领域对抗训练实现，保留模态无关但与AD相关的特定信息，而生成对抗网络则添加不可区分的模态特定风格。 我们在阿尔茨海默病神经影像学倡议（ADNI）研究中评估了我们提出的框架，并将合成数据的质量与其他插补方法进行了比较。我们的生成指标和实际指标之间的平均Cohen's d值小于0.19，这表明无论模态类型如何，合成数据在实践中都是可以使用的。 简而言之： 本文提出了一种利用现有影像数据生成缺失影像测量的方法框架，该框架能够保留阿尔茨海默病特有的信息，并添加不可区分的模态特有风格。通过对ADNI研究中的评估及与其他插补方法的比较，证实了所生成的数据在实际应用中具有可行性与实用性。
URL
https://arxiv.org/abs/2503.02898
PDF
-
IBURD: Image Blending for Underwater Robotic Detection
2025-02-24 22:56:49Jungseok Hong, Sakshi Singh, Junaed SattarAbstract
We present an image blending pipeline, \textit{IBURD}, that creates realistic synthetic images to assist in the training of deep detectors for use on underwater autonomous vehicles (AUVs) for marine debris detection tasks. Specifically, IBURD generates both images of underwater debris and their pixel-level annotations, using source images of debris objects, their annotations, and target background images of marine environments. With Poisson editing and style transfer techniques, IBURD is even able to robustly blend transparent objects into arbitrary backgrounds and automatically adjust the style of blended images using the blurriness metric of target background images. These generated images of marine debris in actual underwater backgrounds address the data scarcity and data variety problems faced by deep-learned vision algorithms in challenging underwater conditions, and can enable the use of AUVs for environmental cleanup missions. Both quantitative and robotic evaluations of IBURD demonstrate the efficacy of the proposed approach for robotic detection of marine debris.
Abstract (translated)
我们提出了一种图像融合流水线\textit{IBURD}，它生成逼真的合成图像，以帮助训练用于海底自主水下航行器（AUV）的海洋垃圾检测任务的深度探测器。具体而言，IBURD 使用物体碎片源图、它们的标注和目标背景图来生成海底垃圾图像及其像素级注释。通过使用泊松编辑和风格转换技术，IBURD 能够将透明对象无缝融合到任意背景中，并且能够自动根据目标背景图的模糊度调整合成图像的风格。这些在实际水下环境中生成的海洋碎片图像可以解决深度学习视觉算法在复杂水下条件下面临的缺乏多样性和数据稀缺性的问题，并能促进AUV用于环境清理任务的应用。IBURD 的定量和机器人测试评估证明了该方法在机器人检测海洋垃圾方面的有效性。
URL
https://arxiv.org/abs/2502.17706
PDF
-
PQDAST: Depth-Aware Arbitrary Style Transfer for Games via Perceptual Quality-Guided Distillation
2025-02-24 09:29:25Eleftherios Ioannou, Steve MaddockAbstract
Artistic style transfer is concerned with the generation of imagery that combines the content of an image with the style of an artwork. In the realm of computer games, most work has focused on post-processing video frames. Some recent work has integrated style transfer into the game pipeline, but it is limited to single styles. Integrating an arbitrary style transfer method into the game pipeline is challenging due to the memory and speed requirements of games. We present PQDAST, the first solution to address this. We use a perceptual quality-guided knowledge distillation framework and train a compressed model using the FLIP evaluator, which substantially reduces both memory usage and processing time with limited impact on stylisation quality. For better preservation of depth and fine details, we utilise a synthetic dataset with depth and temporal considerations during training. The developed model is injected into the rendering pipeline to further enforce temporal stability and avoid diminishing post-process effects. Quantitative and qualitative experiments demonstrate that our approach achieves superior performance in temporal consistency, with comparable style transfer quality, to state-of-the-art image, video and in-game methods.
Abstract (translated)
艺术风格转换旨在生成结合了图像内容与艺术品风格的影像。在计算机游戏中，大多数工作集中在对视频帧进行后处理上。尽管一些近期的研究尝试将风格转移集成到游戏流水线中，但这些方法仅限于单一风格的应用。由于游戏对内存和速度的需求较高，因此将任意风格转换方法融入游戏流水线是一个挑战性的课题。我们提出了PQDAST（感知质量引导的知识蒸馏框架），这是首个解决这一问题的方案。 通过使用感知质量指导的知识蒸馏框架，并利用FLIP评估器训练一个压缩模型，我们的方法显著降低了内存消耗和处理时间，同时对风格化效果的影响有限。为了更好地保留深度信息与细节特征，在训练过程中我们采用了一个合成数据集，并考虑了深度和时间因素。开发的模型被注入到渲染流水线中，以进一步增强时间稳定性并避免减弱后期处理效果。 通过定量和定性的实验验证，我们的方法在时间一致性上表现出了优越性，同时风格转移质量与现有图像、视频及游戏内方法相当甚至更优。
URL
https://arxiv.org/abs/2502.16996
PDF
-
mStyleDistance: Multilingual Style Embeddings and their Evaluation
2025-02-21 03:11:41Justin Qiu, Jiacheng Zhu, Ajay Patel, Marianna Apidianaki, Chris Callison-BurchAbstract
Style embeddings are useful for stylistic analysis and style transfer; however, only English style embeddings have been made available. We introduce Multilingual StyleDistance (mStyleDistance), a multilingual style embedding model trained using synthetic data and contrastive learning. We train the model on data from nine languages and create a multilingual STEL-or-Content benchmark (Wegmann et al., 2022) that serves to assess the embeddings' quality. We also employ our embeddings in an authorship verification task involving different languages. Our results show that mStyleDistance embeddings outperform existing models on these multilingual style benchmarks and generalize well to unseen features and languages. We make our model publicly available at this https URL .
Abstract (translated)
风格嵌入在风格分析和风格转换中很有用；然而，只有英语的风格嵌入被公开提供。我们引入了多语言StyleDistance（mStyleDistance），这是一个通过合成数据和对比学习训练的多语言风格嵌入模型。我们在九种语言的数据上对模型进行了训练，并建立了一个服务于评估嵌入质量的多语言STEL或内容基准（Wegmann等，2022）。我们还将在涉及不同语言的作者身份验证任务中使用我们的嵌入。结果表明，在这些多语言风格基准测试中，mStyleDistance嵌入超过了现有模型的表现，并且很好地泛化到未见过的特性和语言上。我们在[此处](https://此URL)公开发布了该模型。
URL
https://arxiv.org/abs/2502.15168
PDF
-
A Meta-Evaluation of Style and Attribute Transfer Metrics
2025-02-20 20:16:34Amalie Brogaard Pauli, Isabelle Augenstein, Ira AssentAbstract
LLMs make it easy to rewrite text in any style, be it more polite, persuasive, or more positive. We present a large-scale study of evaluation metrics for style and attribute transfer with a focus on content preservation; meaning content not attributed to the style shift is preserved. The de facto evaluation approach uses lexical or semantic similarity metrics often between source sentences and rewrites. While these metrics are not designed to distinguish between style or content differences, empirical meta-evaluation shows a reasonable correlation to human judgment. In fact, recent works find that LLMs prompted as evaluators are only comparable to semantic similarity metrics, even though intuitively, the LLM approach should better fit the task. To investigate this discrepancy, we benchmark 8 metrics for evaluating content preservation on existing datasets and additionally construct a new test set that better aligns with the meta-evaluation aim. Indeed, we then find that the empirical conclusion aligns with the intuition: content preservation metrics for style/attribute transfer must be conditional on the style shift. To support this, we propose a new efficient zero-shot evaluation method using the likelihood of the next token. We hope our meta-evaluation can foster more research on evaluating content preservation metrics, and also to ensure fair evaluation of methods for conducting style transfer.
Abstract (translated)
大型语言模型（LLMs）使得以任何风格重写文本变得容易，无论是更加礼貌、更具说服力还是更积极。我们提出了一项大规模的研究，评估用于风格和属性转换的评价指标，重点关注内容保留；也就是说，在不涉及风格变化的情况下，内容被保持不变。目前的事实标准评估方法使用词汇或语义相似性度量，通常是在源句子与重写后的句子之间进行比较。尽管这些度量不是为区分风格或内容差异而设计的，但经验元评价显示它们与人类判断有相当合理的相关性。事实上，最近的研究发现，作为评估者的LLMs在性能上仅相当于语义相似性度量，虽然从直觉上看，LLM的方法应该更适合这一任务。为了调查这种不一致的原因，我们在现有的数据集上对8种用于评估内容保留的指标进行了基准测试，并额外构建了一个新的测试集以更好地与元评价目标相符合。确实，在此之后我们发现经验结论与直觉相符：风格/属性转换的内容保留度量必须基于风格转变条件化。为此，我们提出了一种使用下一个标记可能性的新零样本评估方法，该方法更为高效。我们希望我们的元评价可以促进更多关于内容保留指标的评估研究，并确保对进行样式转换的方法进行公平评估。
URL
https://arxiv.org/abs/2502.15022
PDF
-
PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data
2025-02-20 09:35:38Shijie Huang, Yiren Song, Yuxuan Zhang, Hailong Guo, Xueyin Wang, Mike Zheng Shou, Jiaming LiuAbstract
We introduce PhotoDoodle, a novel image editing framework designed to facilitate photo doodling by enabling artists to overlay decorative elements onto photographs. Photo doodling is challenging because the inserted elements must appear seamlessly integrated with the background, requiring realistic blending, perspective alignment, and contextual coherence. Additionally, the background must be preserved without distortion, and the artist's unique style must be captured efficiently from limited training data. These requirements are not addressed by previous methods that primarily focus on global style transfer or regional inpainting. The proposed method, PhotoDoodle, employs a two-stage training strategy. Initially, we train a general-purpose image editing model, OmniEditor, using large-scale data. Subsequently, we fine-tune this model with EditLoRA using a small, artist-curated dataset of before-and-after image pairs to capture distinct editing styles and techniques. To enhance consistency in the generated results, we introduce a positional encoding reuse mechanism. Additionally, we release a PhotoDoodle dataset featuring six high-quality styles. Extensive experiments demonstrate the advanced performance and robustness of our method in customized image editing, opening new possibilities for artistic creation.
Abstract (translated)
我们介绍了PhotoDoodle，这是一个新颖的图像编辑框架，旨在通过允许艺术家将装饰性元素叠加到照片上来促进照片涂鸦。然而，由于插入的元素必须与背景无缝融合，这使得照片涂鸦具有挑战性，需要实现逼真的混合、透视对齐和语境一致性。此外，还需要保持背景不被扭曲，并且要高效地捕捉艺术家独特的风格，即使训练数据量有限也是如此。这些需求并未被以往主要关注全局样式转换或局部修复的方法所解决。 我们的方法PhotoDoodle采用了一个两阶段的训练策略。首先，我们使用大规模的数据集来训练一个通用目的图像编辑模型OmniEditor。然后，我们通过EditLoRA微调该模型，并使用艺术家精心挑选的小规模“前”、“后”图像对数据集来捕捉独特的编辑风格和技巧。为了提高生成结果的一致性，我们引入了一种位置编码重用机制。 此外，我们发布了包含六种高质量样式的PhotoDoodle数据集。广泛的实验表明了我们的方法在定制化图像编辑中的高级性能和鲁棒性，为艺术创作开辟了新的可能性。
URL
https://arxiv.org/abs/2502.14397
PDF
-
Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMs
2025-02-18 16:11:54Zixiao Wang, Duzhen Zhang, Ishita Agrawal, Shen Gao, Le Song, Xiuying ChenAbstract
Previous approaches to persona simulation large language models (LLMs) have typically relied on learning basic biographical information, or using limited role-play dialogue datasets to capture a character's responses. However, a holistic representation of an individual goes beyond surface-level facts or conversations to deeper thoughts and thinking. In this work, we introduce CharacterBot, a model designed to replicate both the linguistic patterns and distinctive thought processes of a character. Using Lu Xun, a renowned Chinese writer, as a case study, we propose four training tasks derived from his 17 essay collections. These include a pre-training task focused on mastering external linguistic structures and knowledge, as well as three fine-tuning tasks: multiple-choice question answering, generative question answering, and style transfer, each aligning the LLM with Lu Xun's internal ideation and writing style. To optimize learning across these tasks, we introduce a CharLoRA parameter updating mechanism, where a general linguistic style expert collaborates with other task-specific experts to better study both the language style and the understanding of deeper thoughts. We evaluate CharacterBot on three tasks for linguistic accuracy and opinion comprehension, demonstrating that it significantly outperforms the baselines on our adapted metrics. We hope that this work inspires future research on deep character persona simulation LLM.
Abstract (translated)
此前，用于人格模拟的大语言模型（LLM）的方法通常依赖于学习基本的传记信息或使用有限的角色扮演对话数据集来捕捉角色的响应。然而，一个人物的全面表示超越了表面事实或对话，深入到更深层次的思想和思维方式中。在本研究中，我们介绍了CharacterBot，这是一个旨在复制人物的语言模式及其独特思维过程的模型。以中国著名作家鲁迅为案例研究，我们提出了四项训练任务，这些任务源自他的17部散文集。这包括一个预训练任务，专注于掌握外部语言结构和知识，以及三个微调任务：多选题问答、生成式问题回答和风格转换，每个任务都旨在将LLM与鲁迅的内部思考方式和写作风格对齐。为了在这四项任务中优化学习过程，我们引入了CharLoRA参数更新机制，在这个机制中，一个通用的语言风格专家与其他特定任务的专家合作，以更好地研究语言风格以及更深层次思想的理解。我们在三个任务上评估CharacterBot的语言准确性和观点理解能力，并证明它在我们的自适应指标上显著优于基线模型。我们希望这项工作能激发未来关于深度人物角色模拟大语言模型的研究。
URL
https://arxiv.org/abs/2502.12988
PDF
-
Image Inversion: A Survey from GANs to Diffusion and Beyond
2025-02-17 16:20:48Yinan Chen, Jiangning Zhang, Yali Bi, Xiaobin Hu, Teng Hu, Zhucun Xue, Ran Yi, Yong Liu, Ying TaiAbstract
Image inversion is a fundamental task in generative models, aiming to map images back to their latent representations to enable downstream applications such as editing, restoration, and style transfer. This paper provides a comprehensive review of the latest advancements in image inversion techniques, focusing on two main paradigms: Generative Adversarial Network (GAN) inversion and diffusion model inversion. We categorize these techniques based on their optimization methods. For GAN inversion, we systematically classify existing methods into encoder-based approaches, latent optimization approaches, and hybrid approaches, analyzing their theoretical foundations, technical innovations, and practical trade-offs. For diffusion model inversion, we explore training-free strategies, fine-tuning methods, and the design of additional trainable modules, highlighting their unique advantages and limitations. Additionally, we discuss several popular downstream applications and emerging applications beyond image tasks, identifying current challenges and future research directions. By synthesizing the latest developments, this paper aims to provide researchers and practitioners with a valuable reference resource, promoting further advancements in the field of image inversion. We keep track of the latest works at this https URL
Abstract (translated)
图像反转是生成模型中的一个基本任务，其目标是将图像映射回它们的潜在表示形式，以便实现下游应用如编辑、恢复和风格转换。本文对最新的图像反转技术进展进行了全面回顾，并重点介绍了两大主要范式：生成对抗网络（GAN）反转和扩散模型反转。我们根据优化方法对这些技术进行分类。 对于GAN反转，我们将现有方法系统地分为基于编码器的方法、潜在空间优化方法和混合方法，分析它们的理论基础、技术创新及实际权衡。对于扩散模型反转，本文探索了无训练策略、微调方法以及额外可训练模块的设计，并突出了它们的独特优势和局限性。 此外，我们还讨论了几种流行的下游应用以及其他超越图像任务的新颖应用，识别当前面临的挑战并指出未来的研究方向。通过综合最新的发展动态，本论文旨在为研究者和实践者提供有价值的参考资源，推动该领域的进一步进步。最新工作的跟踪可以在提供的链接中找到：[此处插入具体URL]
URL
https://arxiv.org/abs/2502.11974
PDF
-
Unpaired Image-to-Image Translation with Content Preserving Perspective: A Review
2025-02-11 20:09:29Mehran Safayani, Behnaz Mirzapour, Hanieh Aghaebrahimian, Nasrin Salehi, Hamid RavaeeAbstract
Image-to-image translation (I2I) transforms an image from a source domain to a target domain while preserving source content. Most computer vision applications are in the field of image-to-image translation, such as style transfer, image segmentation, and photo enhancement. The degree of preservation of the content of the source images in the translation process can be different according to the problem and the intended application. From this point of view, in this paper, we divide the different tasks in the field of image-to-image translation into three categories: Fully Content preserving, Partially Content preserving, and Non-Content preserving. We present different tasks, datasets, methods, results of methods for these three categories in this paper. We make a categorization for I2I methods based on the architecture of different models and study each category separately. In addition, we introduce well-known evaluation criteria in the I2I translation field. Specifically, nearly 70 different I2I models were analyzed, and more than 10 quantitative evaluation metrics and 30 distinct tasks and datasets relevant to the I2I translation problem were both introduced and assessed. Translating from simulation to real images could be well viewed as an application of fully content preserving or partially content preserving unsupervised image-to-image translation methods. So, we provide a benchmark for Sim-to-Real translation, which can be used to evaluate different methods. In general, we conclude that because of the different extent of the obligation to preserving content in various applications, it is better to consider this issue in choosing a suitable I2I model for a specific application.
Abstract (translated)
图像到图像翻译（I2I）是指将一个领域中的图像转换为另一个领域的图像，同时保留源图的内容。大多数计算机视觉应用都属于图像到图像翻译的范畴，例如风格迁移、图像分割和照片增强等。在翻译过程中，根据问题类型及应用场景的不同，保留源图内容的程度也会有所差异。从这个角度来看，在本文中我们将不同任务分为三类：完全保留内容（Fully Content Preserving）、部分保留内容（Partially Content Preserving）以及不保留内容（Non-Content Preserving）。在这篇文章中，我们介绍了这三种类别下的不同任务、数据集、方法及其相应的方法结果。基于模型的不同架构，我们对I2I方法进行了分类，并单独研究了每个类别。此外，还介绍了一些在图像到图像翻译领域内广为认可的评估标准。具体来说，本文分析了近70种不同的I2I模型，并介绍了超过10项定量评估指标及30个与I2I翻译问题相关的不同任务和数据集。 从模拟图像向真实图像转换可以视作完全保留内容或部分保留内容的无监督图像到图像翻译方法的应用。因此，我们提供了Sim-to-Real（仿真到现实）翻译的一个基准测试平台，可用于评估不同的方法。总的来说，我们认为由于各种应用在保留内容方面的义务程度不同，选择适用于特定应用场景的I2I模型时应该考虑这个问题。
URL
https://arxiv.org/abs/2502.08667
PDF
-
Direct Ascent Synthesis: Revealing Hidden Generative Capabilities in Discriminative Models
2025-02-11 18:27:27Stanislav Fort, Jonathan WhitakerAbstract
We demonstrate that discriminative models inherently contain powerful generative capabilities, challenging the fundamental distinction between discriminative and generative architectures. Our method, Direct Ascent Synthesis (DAS), reveals these latent capabilities through multi-resolution optimization of CLIP model representations. While traditional inversion attempts produce adversarial patterns, DAS achieves high-quality image synthesis by decomposing optimization across multiple spatial scales (1x1 to 224x224), requiring no additional training. This approach not only enables diverse applications -- from text-to-image generation to style transfer -- but maintains natural image statistics ($1/f^2$ spectrum) and guides the generation away from non-robust adversarial patterns. Our results demonstrate that standard discriminative models encode substantially richer generative knowledge than previously recognized, providing new perspectives on model interpretability and the relationship between adversarial examples and natural image synthesis.
Abstract (translated)
我们展示了判别模型本质上含有强大的生成能力，这挑战了判别架构和生成架构之间的基本区别。我们的方法，直接上升合成（Direct Ascent Synthesis, DAS），通过在CLIP模型表示上的多分辨率优化揭示这些潜在的能力。虽然传统的逆向尝试会产生对抗性模式，但DAS通过将优化分解到多个空间尺度（从1x1到224x224）实现了高质量的图像合成，并且无需额外训练。这种方法不仅能够支持多种应用——包括文本到图像生成和风格迁移——还能保持自然图像统计特性（$1/f^2$频谱），并指导生成远离非鲁棒性的对抗性模式。我们的结果表明，标准判别模型编码了比之前认识到的更为丰富的生成知识，这为模型可解释性和对抗样本与自然图像合成之间的关系提供了新的视角。
URL
https://arxiv.org/abs/2502.07753
PDF
-
Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models
2025-02-11 11:17:39Lin Zhu, Xinbing Wang, Chenghu Zhou, Qinying Gu, Nanyang YeAbstract
Given a style-reference image as the additional image condition, text-to-image diffusion models have demonstrated impressive capabilities in generating images that possess the content of text prompts while adopting the visual style of the reference image. However, current state-of-the-art methods often struggle to disentangle content and style from style-reference images, leading to issues such as content leakages. To address this issue, we propose a masking-based method that efficiently decouples content from style without the need of tuning any model parameters. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle: guiding with appropriately-selected fewer conditions (e.g., dropping several image feature elements) can efficiently avoid unwanted content flowing into the diffusion models, enhancing the style transfer performances of text-to-image diffusion models. In this paper, we validate this finding both theoretically and experimentally. Extensive experiments across various styles demonstrate the effectiveness of our masking-based method and support our theoretical results.
Abstract (translated)
在给定风格参考图像作为附加条件的情况下，文本到图像的扩散模型展示出了生成同时具备文本提示内容并采用参考图像视觉风格的图片的强大能力。然而，目前最先进的方法常常难以从风格参考图中分离出内容和风格，导致诸如内容泄露等问题的发生。为解决这一问题，我们提出了一种基于掩码的方法，能够在不调整任何模型参数的情况下有效分离内容与风格。通过简单地在风格参考图像特征的特定元素上进行掩码处理，我们揭示了一个重要但尚未充分探索的原则：使用适当选择的较少条件（例如，删除几个图像特征元素）可以有效地避免不必要的内容流入扩散模型，从而提升文本到图像扩散模型的风格迁移性能。本文中，我们在理论和实验两方面验证了这一发现的有效性。跨多种风格进行的大量实验展示了我们基于掩码的方法的有效性，并支持我们的理论结果。
URL
https://arxiv.org/abs/2502.07466
PDF
-
Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model
2025-02-08 07:14:04Jialong Zuo, Shengpeng Ji, Minghui Fang, Ziyue Jiang, Xize Cheng, Qian Yang, Wenrui Liu, Guangyan Zhang, Zehai Tu, Yiwen Guo, Zhou ZhaoAbstract
This paper introduces PFlow-VC, a conditional flow matching voice conversion model that leverages fine-grained discrete pitch tokens and target speaker prompt information for expressive voice conversion (VC). Previous VC works primarily focus on speaker conversion, with further exploration needed in enhancing expressiveness (such as prosody and emotion) for timbre conversion. Unlike previous methods, we adopt a simple and efficient approach to enhance the style expressiveness of voice conversion models. Specifically, we pretrain a self-supervised pitch VQVAE model to discretize speaker-irrelevant pitch information and leverage a masked pitch-conditioned flow matching model for Mel-spectrogram synthesis, which provides in-context pitch modeling capabilities for the speaker conversion model, effectively improving the voice style transfer capacity. Additionally, we improve timbre similarity by combining global timbre embeddings with time-varying timbre tokens. Experiments on unseen LibriTTS test-clean and emotional speech dataset ESD show the superiority of the PFlow-VC model in both timbre conversion and style transfer. Audio samples are available on the demo page this https URL.
Abstract (translated)
这篇论文介绍了PFlow-VC，这是一种条件流匹配声音转换模型，利用细粒度的离散音高令牌和目标说话人的提示信息来进行富有表现力的声音转换（Voice Conversion, VC）。之前的声音转换工作主要集中在说话人转换上，在提升音色转换中的表达性方面（如韵律和情感）还有待进一步探索。与以往的方法不同，我们采用了一种简单且高效的方式来增强声音转换模型的风格表达能力。具体来说，我们预先训练了一个自我监督的音高VQVAE模型来离散化与说话人无关的音高信息，并利用一个遮蔽音高标准匹配流合成模型进行梅尔频谱图生成，这为说话人转换模型提供了上下文中的音高建模能力，有效地提升了语音风格传输的能力。此外，我们通过结合全局音色嵌入与时变音色调料来改进音色相似度。 在未见过的LibriTTS测试集（test-clean）和情感语音数据集ESD上的实验表明，PFlow-VC模型在音色转换和风格迁移方面均表现出优越性。音频样本可在该演示页面上找到：[此URL]（请将方括号中的文本替换为实际提供的链接）。
URL
https://arxiv.org/abs/2502.05471
PDF
-
Coarse-to-Fine Structure-Aware Artistic Style Transfer
2025-02-08 00:04:12Kunxiao Liu, Guowu Yuan, Hao Wu, Wenhua QianAbstract
Artistic style transfer aims to use a style image and a content image to synthesize a target image that retains the same artistic expression as the style image while preserving the basic content of the content image. Many recently proposed style transfer methods have a common problem; that is, they simply transfer the texture and color of the style image to the global structure of the content image. As a result, the content image has a local structure that is not similar to the local structure of the style image. In this paper, we present an effective method that can be used to transfer style patterns while fusing the local style structure into the local content structure. In our method, dif-ferent levels of coarse stylized features are first reconstructed at low resolution using a Coarse Network, in which style color distribution is roughly transferred, and the content structure is combined with the style structure. Then, the reconstructed features and the content features are adopted to synthesize high-quality structure-aware stylized images with high resolution using a Fine Network with three structural selective fusion (SSF) modules. The effectiveness of our method is demonstrated through the generation of appealing high-quality stylization results and a com-parison with some state-of-the-art style transfer methods.
Abstract (translated)
艺术风格转换的目标是利用一幅风格图像和一幅内容图像来合成目标图像，使其保留与风格图像相同的艺术表达的同时保持内容图像的基本内容不变。然而，许多最近提出的风格转移方法存在一个共同的问题：它们只是简单地将风格图像的纹理和颜色转移到内容图像的整体结构上。因此，生成的内容图像在局部结构上往往并不类似风格图像的局部结构。 在这篇论文中，我们提出了一种有效的方法，可以用于在融合局部风格结构到局部内容结构的同时进行样式图案转移。我们的方法首先使用一个粗略网络（Coarse Network）在低分辨率下重构不同级别的粗略风格化特征，在此过程中大致传输了风格颜色分布，并结合了内容与风格的结构。然后，通过一个配备有三个结构选择性融合（SSF）模块的精细网络（Fine Network），利用重建后的特征和内容特征来生成高分辨率且具有高度感知结构意识的风格化图像。 我们的方法的有效性通过生成吸引人的高质量风格化结果以及与其他最新风格转移方法进行比较得到了证明。
URL
https://arxiv.org/abs/2502.05387
PDF
