Style_Transfer
-
Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning
2025-05-21 20:37:33Qiang Zhu, Kuan Lu, Menghao Huo, Yuxiao LiAbstract
Image-to-image translation aims to learn a mapping between a source and a target domain, enabling tasks such as style transfer, appearance transformation, and domain adaptation. In this work, we explore a diffusion-based framework for image-to-image translation by adapting Diffusion Transformers (DiT), which combine the denoising capabilities of diffusion models with the global modeling power of transformers. To guide the translation process, we condition the model on image embeddings extracted from a pre-trained CLIP encoder, allowing for fine-grained and structurally consistent translations without relying on text or class labels. We incorporate both a CLIP similarity loss to enforce semantic consistency and an LPIPS perceptual loss to enhance visual fidelity during training. We validate our approach on two benchmark datasets: face2comics, which translates real human faces to comic-style illustrations, and edges2shoes, which translates edge maps to realistic shoe images. Experimental results demonstrate that DiT, combined with CLIP-based conditioning and perceptual similarity objectives, achieves high-quality, semantically faithful translations, offering a promising alternative to GAN-based models for paired image-to-image translation tasks.
Abstract (translated)
图像到图像翻译旨在学习源域和目标域之间的映射，从而实现如风格转换、外观变换和领域适应等任务。在这项工作中，我们探索了一种基于扩散框架的图像到图像翻译方法，通过调整 Diffusion Transformers（DiT）来实现这一目的。Diffusion Transformers 结合了扩散模型的去噪能力与变压器的整体建模能力。为了引导翻译过程，我们将模型条件化设置为从预训练 CLIP 编码器中提取的图像嵌入，从而使翻译更加精细且结构一致，并且无需依赖文本或类别标签。我们引入了一种基于 CLIP 的相似性损失来强制执行语义一致性以及一种 LPIPS 视觉感知损失以在训练期间增强视觉保真度。 我们在两个基准数据集上验证了我们的方法：face2comics，该数据集将真实的人脸翻译成漫画风格的插图；edges2shoes，该数据集则将边缘地图转换为逼真的鞋类图像。实验结果表明，DiT 结合 CLIP 基础条件化和感知相似性目标，能够实现高质量且语义忠实的翻译，这为配对图像到图像翻译任务提供了一种有前景的GAN模型替代方案。
URL
https://arxiv.org/abs/2505.16001
PDF
-
GT^2-GS: Geometry-aware Texture Transfer for Gaussian Splatting
2025-05-21 07:37:29Wenjie Liu, Zhongliang Liu, Junwei Shu, Changbo Wang, Yang LiAbstract
Transferring 2D textures to 3D modalities is of great significance for improving the efficiency of multimedia content creation. Existing approaches have rarely focused on transferring image textures onto 3D representations. 3D style transfer methods are capable of transferring abstract artistic styles to 3D scenes. However, these methods often overlook the geometric information of the scene, which makes it challenging to achieve high-quality 3D texture transfer results. In this paper, we present GT^2-GS, a geometry-aware texture transfer framework for gaussian splitting. From the perspective of matching texture features with geometric information in rendered views, we identify the issue of insufficient texture features and propose a geometry-aware texture augmentation module to expand the texture feature set. Moreover, a geometry-consistent texture loss is proposed to optimize texture features into the scene representation. This loss function incorporates both camera pose and 3D geometric information of the scene, enabling controllable texture-oriented appearance editing. Finally, a geometry preservation strategy is introduced. By alternating between the texture transfer and geometry correction stages over multiple iterations, this strategy achieves a balance between learning texture features and preserving geometric integrity. Extensive experiments demonstrate the effectiveness and controllability of our method. Through geometric awareness, our approach achieves texture transfer results that better align with human visual perception. Our homepage is available at this https URL.
Abstract (translated)
将2D纹理转移到3D模态对于提升多媒体内容创建的效率具有重要意义。现有方法很少关注如何将图像纹理映射到3D表示上。虽然现有的3D风格迁移方法能够把抽象的艺术风格转移到3D场景中，但这些方法常常忽略了场景的几何信息，这使得实现高质量的3D纹理转移变得困难。 在本文中，我们提出了一种基于高斯分割的、具有几何感知能力的纹理传输框架（GT^2-GS）。从匹配渲染视图中的纹理特征与几何信息的角度出发，我们发现了现有方法中存在的纹理特征不足的问题，并提出了一个增强几何感知的纹理扩展模块来扩充纹理特征集。此外，还提出了一种基于几何一致性的纹理损失函数，用以优化场景表示中的纹理特征。这一损失函数结合了相机姿态和3D场景的几何信息，支持对目标外观进行可控制的编辑。最后，我们引入了一个几何保持策略，在多个迭代中通过交替执行纹理转移与几何校正阶段来平衡学习到的纹理特征和保持几何完整性的关系。 广泛的实验结果证明了我们方法的有效性和可控性。通过增强对场景几何信息的理解，我们的方法实现了更好地符合人类视觉感知的纹理传输效果。 如需进一步了解，请访问我们的主页：[这里插入URL]
URL
https://arxiv.org/abs/2505.15208
PDF
-
TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis
2025-05-20 21:04:10Yu Zhang, Wenxiang Guo, Changhao Pan, Dongyu Yao, Zhiyuan Zhu, Ziyue Jiang, Yuhan Wang, Tao Jin, Zhou ZhaoAbstract
Customizable multilingual zero-shot singing voice synthesis (SVS) has various potential applications in music composition and short video dubbing. However, existing SVS models overly depend on phoneme and note boundary annotations, limiting their robustness in zero-shot scenarios and producing poor transitions between phonemes and notes. Moreover, they also lack effective multi-level style control via diverse prompts. To overcome these challenges, we introduce TCSinger 2, a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts. TCSinger 2 mainly includes three key modules: 1) Blurred Boundary Content (BBC) Encoder, predicts duration, extends content embedding, and applies masking to the boundaries to enable smooth transitions. 2) Custom Audio Encoder, uses contrastive learning to extract aligned representations from singing, speech, and textual prompts. 3) Flow-based Custom Transformer, leverages Cus-MOE, with F0 supervision, enhancing both the synthesis quality and style modeling of the generated singing voice. Experimental results show that TCSinger 2 outperforms baseline models in both subjective and objective metrics across multiple related tasks.
Abstract (translated)
可定制的多语言零样本歌声合成（SVS）在音乐创作和短视频配音中具有多种潜在应用。然而，现有的SVS模型过于依赖音素和音符边界的标注，这限制了它们在零样本场景中的鲁棒性，并且导致音素和音符之间的过渡效果不佳。此外，这些模型还缺乏通过多样化提示进行有效的多层次风格控制的能力。 为了解决这些问题，我们引入了TCSinger 2，这是一个基于多种提示的多任务、跨语言零样本歌声合成模型，具备风格转换和风格控制功能。TCSinger 2主要包括三个关键模块： 1. **模糊边界内容（BBC）编码器**：预测时长，扩展内容嵌入，并在边界的上下文应用掩码以实现平滑过渡。 2. **定制音频编码器**：利用对比学习从歌唱、言语和文本提示中提取对齐的表示，增强模型的学习能力与泛化性。 3. **基于流的自定义变压器**：采用Cus-MOE，并通过基频（F0）监督来提高生成歌声的质量和风格建模能力。 实验结果显示，在多个相关任务上，TCSinger 2在主观和客观指标方面均优于基准模型。
URL
https://arxiv.org/abs/2505.14910
PDF
-
Handloom Design Generation Using Generative Networks
2025-05-20 13:16:55Rajat Kanti Bhattacharjee, Meghali Nandi, Amrit Jha, Gunajit Kalita, Ferdous Ahmed BarbhuiyaAbstract
This paper proposes deep learning techniques of generating designs for clothing, focused on handloom fabric and discusses the associated challenges along with its application. The capability of generative neural network models in understanding artistic designs and synthesizing those is not yet explored well. In this work, multiple methods are employed incorporating the current state of the art generative models and style transfer algorithms to study and observe their performance for the task. The results are then evaluated through user score. This work also provides a new dataset NeuralLoom for the task of the design generation.
Abstract (translated)
本文提出了一种用于生成服装设计的深度学习技术，重点是手织布料，并讨论了相关的挑战及其应用。生成式神经网络模型在理解和合成艺术设计方面的能力尚未得到充分探索。在这项工作中，采用了多种方法，结合当前最先进的生成模型和风格转换算法来研究并观察它们在此任务中的性能表现。通过用户评分对结果进行评估。此外，本文还提供了一个新的数据集 NeuralLoom 用于设计生成任务。
URL
https://arxiv.org/abs/2505.14330
PDF
-
OmniStyle: Filtering High Quality Style Transfer Data at Scale
2025-05-20 07:29:21Ye Wang, Ruiqi Liu, Jiang Lin, Fei Liu, Zili Yi, Yilin Wang, Rui MaAbstract
In this paper, we introduce OmniStyle-1M, a large-scale paired style transfer dataset comprising over one million content-style-stylized image triplets across 1,000 diverse style categories, each enhanced with textual descriptions and instruction prompts. We show that OmniStyle-1M can not only enable efficient and scalable of style transfer models through supervised training but also facilitate precise control over target stylization. Especially, to ensure the quality of the dataset, we introduce OmniFilter, a comprehensive style transfer quality assessment framework, which filters high-quality triplets based on content preservation, style consistency, and aesthetic appeal. Building upon this foundation, we propose OmniStyle, a framework based on the Diffusion Transformer (DiT) architecture designed for high-quality and efficient style transfer. This framework supports both instruction-guided and image-guided style transfer, generating high resolution outputs with exceptional detail. Extensive qualitative and quantitative evaluations demonstrate OmniStyle's superior performance compared to existing approaches, highlighting its efficiency and versatility. OmniStyle-1M and its accompanying methodologies provide a significant contribution to advancing high-quality style transfer, offering a valuable resource for the research community.
Abstract (translated)
在这篇论文中，我们介绍了OmniStyle-1M，这是一个大规模的配对风格迁移数据集，包含超过一百万张内容-样式-渲染后的图像三元组，涵盖了1000种多样化的风格类别，并且每一种都配备了文本描述和指令提示。我们展示了OmniStyle-1M不仅可以通过监督训练实现高效可扩展的风格迁移模型开发，还能促进对目标样式的精准控制。特别是为了确保数据集的质量，我们引入了OmniFilter，一个全面的风格迁移质量评估框架，该框架基于内容保留、风格一致性和美学吸引力过滤出高质量的三元组。 在此基础上，我们提出了OmniStyle框架，该框架基于Diffusion Transformer（DiT）架构设计，旨在实现高质量和高效的风格迁移。此框架支持指令引导和图像引导的风格迁移，能够生成高分辨率且细节出色的输出。 广泛的定性和定量评估表明，与现有方法相比，OmniStyle具有显著优越的表现，突显了其效率和灵活性。OmniStyle-1M及其相关方法为推动高质量风格迁移的发展做出了重大贡献，提供了宝贵的资源给研究社区。
URL
https://arxiv.org/abs/2505.14028
PDF
-
A Study on the Refining Handwritten Font by Mixing Font Styles
2025-05-19 08:20:43Avinash Kumar, Kyeolhee Kang, Ammar ul Hassan, Jaeyoung ChoiAbstract
Handwritten fonts have a distinct expressive character, but they are often difficult to read due to unclear or inconsistent handwriting. FontFusionGAN (FFGAN) is a novel method for improving handwritten fonts by combining them with printed fonts. Our method implements generative adversarial network (GAN) to generate font that mix the desirable features of handwritten and printed fonts. By training the GAN on a dataset of handwritten and printed fonts, it can generate legible and visually appealing font images. We apply our method to a dataset of handwritten fonts and demonstrate that it significantly enhances the readability of the original fonts while preserving their unique aesthetic. Our method has the potential to improve the readability of handwritten fonts, which would be helpful for a variety of applications including document creation, letter writing, and assisting individuals with reading and writing difficulties. In addition to addressing the difficulties of font creation for languages with complex character sets, our method is applicable to other text-image-related tasks, such as font attribute control and multilingual font style transfer.
Abstract (translated)
手写字体具有独特的表现力，但由于笔迹不清楚或不一致，它们通常难以阅读。FontFusionGAN（FFGAN）是一种结合手写和印刷字体以改进手写字体的新方法。我们的方法利用生成对抗网络（GAN）来创建融合了手写和印刷字体优点的新型字体。通过在包含手写和印刷字体的数据集上训练GAN，它可以生成清晰且视觉效果吸引人的字体图像。我们将该方法应用于一个手写字体数据集，并证明它显著提升了原始字体的可读性，同时保留其独特的美学风格。 我们的方法有望提高手写字体的易读性，在文档创建、书信书写以及帮助阅读和写作有困难的人士等方面大有用处。此外，除了解决复杂字符集语言中字体创作的难题外，我们的方法还适用于其他与文本图像相关任务，例如控制字体属性及跨多种语言风格转换。
URL
https://arxiv.org/abs/2505.12834
PDF
-
SynDec: A Synthesize-then-Decode Approach for Arbitrary Textual Style Transfer via Large Language Models
2025-05-19 08:03:38Han Sun, Zhen Sun, Zongmin Zhang, Linzhao Jia, Wei Shao, Min ZhangAbstract
Large Language Models (LLMs) are emerging as dominant forces for textual style transfer. However, for arbitrary style transfer, LLMs face two key challenges: (1) considerable reliance on manually-constructed prompts and (2) rigid stylistic biases inherent in LLMs. In this paper, we propose a novel Synthesize-then-Decode (SynDec) approach, which automatically synthesizes high-quality prompts and amplifies their roles during decoding process. Specifically, our approach synthesizes prompts by selecting representative few-shot samples, conducting a four-dimensional style analysis, and reranking the candidates. At LLM decoding stage, the TST effect is amplified by maximizing the contrast in output probabilities between scenarios with and without the synthesized prompt, as well as between prompts and negative samples. We conduct extensive experiments and the results show that SynDec outperforms existing state-of-the-art LLM-based methods on five out of six benchmarks (e.g., achieving up to a 9\% increase in accuracy for modern-to-Elizabethan English transfer). Detailed ablation studies further validate the effectiveness of SynDec.
Abstract (translated)
大型语言模型（LLM）正在成为文本风格转换的主导力量。然而，对于任意风格的文本转换，LLM面临着两个关键挑战：(1) 对于手动构建提示词的高度依赖；(2) LLM本身固有的僵化风格偏见。在这篇论文中，我们提出了一种新颖的“合成-解码”（SynDec）方法，该方法能够自动生成高质量的提示词，并在解码过程中增强它们的作用。具体而言，我们的方法通过选择代表性的小样本、进行四维风格分析以及重新排名候选者来合成提示词。在LLM解码阶段，我们通过最大化有无合成提示和正负样本之间输出概率的对比差异来放大TST（文本样式转换）效应。我们进行了广泛的实验，并且结果显示SynDec在六个基准中的五个上优于现有的最先进的基于LLM的方法（例如，在现代英语到伊丽莎白时代英语的风格转换中，准确率提高了高达9%）。详细的消融研究进一步验证了SynDec的有效性。
URL
https://arxiv.org/abs/2505.12821
PDF
-
Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior
2025-05-16 14:40:31Chin-Yun Yu, Marco A. Mart\'inez-Ram\'irez, Junghyun Koo, Wei-Hsiang Liao, Yuki Mitsufuji, Gy\"orgy FazekasAbstract
Style Transfer with Inference-Time Optimisation (ST-ITO) is a recent approach for transferring the applied effects of a reference audio to a raw audio track. It optimises the effect parameters to minimise the distance between the style embeddings of the processed audio and the reference. However, this method treats all possible configurations equally and relies solely on the embedding space, which can lead to unrealistic or biased results. We address this pitfall by introducing a Gaussian prior derived from a vocal preset dataset, DiffVox, over the parameter space. The resulting optimisation is equivalent to maximum-a-posteriori estimation. Evaluations on vocal effects transfer on the MedleyDB dataset show significant improvements across metrics compared to baselines, including a blind audio effects estimator, nearest-neighbour approaches, and uncalibrated ST-ITO. The proposed calibration reduces parameter mean squared error by up to 33% and matches the reference style better. Subjective evaluations with 16 participants confirm our method's superiority, especially in limited data regimes. This work demonstrates how incorporating prior knowledge in inference time enhances audio effects transfer, paving the way for more effective and realistic audio processing systems.
Abstract (translated)
风格转移与推理时间优化（ST-ITO）是一种最近用于将参考音频的效果转移到原始音频轨道上的方法。该方法通过调整效果参数来最小化处理后的音频的样式嵌入和参考之间的距离。然而，这种方法对所有可能的配置一视同仁，并且仅仅依赖于嵌入空间，这可能导致不现实或有偏见的结果。 为了克服这一缺陷，我们引入了一种基于DiffVox（一个声乐预设数据集）的高斯先验，在参数空间上进行设置。由此产生的优化等同于最大后验估计。在MedleyDB数据集上的声效转移评估显示，与基线方法相比，包括盲音频效果估算器、最近邻方法和未经校准的ST-ITO，我们的方法在多个指标上都有显著改进。所提出的校准减少了参数均方误差高达33%，并且更好地匹配了参考样式。 主观评价测试中，16名参与者确认了我们方法的优越性，在数据量有限的情况下尤其明显。这项工作展示了如何通过在推理时间加入先验知识来增强音频效果转移，并为更有效和现实的音频处理系统铺平道路。
URL
https://arxiv.org/abs/2505.11315
PDF
-
SPAST: Arbitrary Style Transfer with Style Priors via Pre-trained Large-scale Model
2025-05-13 15:54:36Zhanjie Zhang, Quanwei Zhang, Junsheng Luan, Mengyuan Yang, Yun Wang, Lei ZhaoAbstract
Given an arbitrary content and style image, arbitrary style transfer aims to render a new stylized image which preserves the content image's structure and possesses the style image's style. Existing arbitrary style transfer methods are based on either small models or pre-trained large-scale models. The small model-based methods fail to generate high-quality stylized images, bringing artifacts and disharmonious patterns. The pre-trained large-scale model-based methods can generate high-quality stylized images but struggle to preserve the content structure and cost long inference time. To this end, we propose a new framework, called SPAST, to generate high-quality stylized images with less inference time. Specifically, we design a novel Local-global Window Size Stylization Module (LGWSSM)tofuse style features into content features. Besides, we introduce a novel style prior loss, which can dig out the style priors from a pre-trained large-scale model into the SPAST and motivate the SPAST to generate high-quality stylized images with short inference this http URL conduct abundant experiments to verify that our proposed method can generate high-quality stylized images and less inference time compared with the SOTA arbitrary style transfer methods.
Abstract (translated)
给定任意的内容图像和风格图像，任意风格迁移的目标是生成一个新的风格化图像，该图保持内容图像的结构，并具备风格图像的特点。现有的任意风格迁移方法要么基于小模型，要么基于预训练的大规模模型。基于小模型的方法无法生成高质量的风格化图像，会产生伪影和不和谐的图案；而基于预训练大规模模型的方法虽然能够生成高质量的风格化图像，但难以保持内容结构，并且需要较长的时间进行推理。为此，我们提出了一种新的框架——SPAST（Style Prior Assisted Stylization Transfer），以更短的推理时间生成高质量的风格化图像。 具体来说，我们设计了一个新颖的局部-全局窗口大小风格化模块（LGWSSM）来融合风格特征和内容特征。此外，我们还引入了一种新的风格先验损失，可以从预训练的大规模模型中提取出风格先验，并激励SPAST在短时间内生成高质量的风格化图像。 我们进行了大量的实验以验证所提出的方法相较于现有的最先进的任意风格迁移方法能够生成更高质量的风格化图像和减少推理时间。
URL
https://arxiv.org/abs/2505.08695
PDF
-
DFA-CON: A Contrastive Learning Approach for Detecting Copyright Infringement in DeepFake Art
2025-05-13 13:23:52Haroon Wahab, Hassan Ugail, Irfan MehmoodAbstract
Recent proliferation of generative AI tools for visual content creation-particularly in the context of visual artworks-has raised serious concerns about copyright infringement and forgery. The large-scale datasets used to train these models often contain a mixture of copyrighted and non-copyrighted artworks. Given the tendency of generative models to memorize training patterns, they are susceptible to varying degrees of copyright violation. Building on the recently proposed DeepfakeArt Challenge benchmark, this work introduces DFA-CON, a contrastive learning framework designed to detect copyright-infringing or forged AI-generated art. DFA-CON learns a discriminative representation space, posing affinity among original artworks and their forged counterparts within a contrastive learning framework. The model is trained across multiple attack types, including inpainting, style transfer, adversarial perturbation, and cutmix. Evaluation results demonstrate robust detection performance across most attack types, outperforming recent pretrained foundation models. Code and model checkpoints will be released publicly upon acceptance.
Abstract (translated)
近期，用于视觉内容创作的生成式人工智能工具（特别是针对视觉艺术作品）的普及引发了关于版权侵权和伪造的严重担忧。训练这些模型所使用的大型数据集通常包含受版权保护的艺术作品与不受版权保护的作品混合在一起的情况。鉴于生成性模型倾向于记住训练模式，它们在不同程度上容易发生版权侵犯行为。 在此背景下，本工作借鉴了最近提出的DeepfakeArt挑战基准测试，并引入了一种名为DFA-CON的对比学习框架，旨在检测侵权或伪造的人工智能生成的艺术作品。该框架通过构建一个区分性的表示空间，在对比学习框架内促进原始艺术作品与其被伪造版本之间的关联性。 模型在多种攻击类型下进行训练，包括图像修复（inpainting）、风格转换、对抗性扰动和混合剪切（cutmix）。评估结果显示，在大多数攻击类型下，该方法具有强大的检测性能，并且优于最近预训练的基础模型。本研究的代码和模型检查点将在接受后公开发布。
URL
https://arxiv.org/abs/2505.08552
PDF
-
Implementing Long Text Style Transfer with LLMs through Dual-Layered Sentence and Paragraph Structure Extraction and Mapping
2025-05-11 05:53:33Yusen Wu, Xiaotie DengAbstract
This paper addresses the challenge in long-text style transfer using zero-shot learning of large language models (LLMs), proposing a hierarchical framework that combines sentence-level stylistic adaptation with paragraph-level structural coherence. We argue that in the process of effective paragraph-style transfer, to preserve the consistency of original syntactic and semantic information, it is essential to perform style transfer not only at the sentence level but also to incorporate paragraph-level semantic considerations, while ensuring structural coherence across inter-sentential relationships. Our proposed framework, ZeroStylus, operates through two systematic phases: hierarchical template acquisition from reference texts and template-guided generation with multi-granular matching. The framework dynamically constructs sentence and paragraph template repositories, enabling context-aware transformations while preserving inter-sentence logical relationships. Experimental evaluations demonstrate significant improvements over baseline methods, with structured rewriting achieving 6.90 average score compared to 6.70 for direct prompting approaches in tri-axial metrics assessing style consistency, content preservation, and expression quality. Ablation studies validate the necessity of both template hierarchies during style transfer, showing higher content preservation win rate against sentence-only approaches through paragraph-level structural encoding, as well as direct prompting method through sentence-level pattern extraction and matching. The results establish new capabilities for coherent long-text style transfer without requiring parallel corpora or LLM fine-tuning.
Abstract (translated)
本文探讨了使用大型语言模型（LLM）的零样本学习在长文本风格转换中的挑战，并提出了一种分层框架，该框架结合了句子级别的风格适应与段落级别的结构一致性。我们主张，在进行有效的段落级风格转换过程中，为了保持原始句法和语义信息的一致性，仅在句子级别上执行风格转移是不够的，还必须纳入段落级别的语义考虑，并确保跨句子关系中的结构连贯性。 我们的框架ZeroStylus通过两个系统阶段运作：从参考文本中获得分层模板以及使用多粒度匹配进行模板引导生成。该框架动态构建了句级和段落级模板库，使得在保持句子间逻辑关系的同时能够进行上下文感知的转换。 实验评估显示，在评估风格一致性、内容保留和表达质量三个维度上的三轴向指标中，结构化重写方法比直接提示方法平均得分高出0.2分（6.90对6.70），超过了基线方法。消融研究验证了在风格转移过程中使用模板层次结构的必要性：通过段落级别的结构性编码，在内容保留胜率上超越仅处理句子的方法；通过句级模式提取和匹配，也优于直接提示法。 这些结果为无需平行语料库或LLM微调的一致长文本风格转换建立了新的能力。
URL
https://arxiv.org/abs/2505.07888
PDF
-
Semantic Style Transfer for Enhancing Animal Facial Landmark Detection
2025-05-08 20:48:15Anadil Hussein, Anna Zamansky, George MartvelAbstract
Neural Style Transfer (NST) is a technique for applying the visual characteristics of one image onto another while preserving structural content. Traditionally used for artistic transformations, NST has recently been adapted, e.g., for domain adaptation and data augmentation. This study investigates the use of this technique for enhancing animal facial landmark detectors training. As a case study, we use a recently introduced Ensemble Landmark Detector for 48 anatomical cat facial landmarks and the CatFLW dataset it was trained on, making three main contributions. First, we demonstrate that applying style transfer to cropped facial images rather than full-body images enhances structural consistency, improving the quality of generated images. Secondly, replacing training images with style-transferred versions raised challenges of annotation misalignment, but Supervised Style Transfer (SST) - which selects style sources based on landmark accuracy - retained up to 98% of baseline accuracy. Finally, augmenting the dataset with style-transferred images further improved robustness, outperforming traditional augmentation methods. These findings establish semantic style transfer as an effective augmentation strategy for enhancing the performance of facial landmark detection models for animals and beyond. While this study focuses on cat facial landmarks, the proposed method can be generalized to other species and landmark detection models.
Abstract (translated)
神经风格迁移（NST）是一种将一幅图像的视觉特征应用于另一幅图象以保留结构内容的技术。这种技术传统上用于艺术转换，但最近已被改编用于领域适应和数据增强等领域。本研究探讨了该技术在提升动物面部标志检测器训练效果中的应用。作为案例研究，我们使用了一种为48个解剖学猫面部标志设计的集合标志探测器以及其基于CatFLW数据集进行的训练，并做出三项主要贡献。 首先，研究表明对裁剪后的面部图像而非全身图像应用风格迁移可以增强结构一致性，从而提高生成图像的质量。其次，在用经过风格转换的图像替换训练图像时出现了注释不一致的问题，但通过监督式风格传输（SST）——一种基于标志准确性选择样式来源的方法——能够保留高达98%的基础准确率。最后，将数据集与经过风格迁移的图像进行扩充进一步提升了模型的鲁棒性，并超越了传统的增强方法。 这些发现确立了语义风格迁移作为一种有效的增强策略，可以用于提升动物面部标志检测模型及其他相关领域的性能表现。虽然本研究重点是猫面部标志，但提出的方法也可以推广到其他物种和标志检测模型上。
URL
https://arxiv.org/abs/2505.05640
PDF
-
Does CLIP perceive art the same way we do?
2025-05-08 13:21:10Andrea Asperti, Leonardo Dess\`i, Maria Chiara Tonetti, Nico WuAbstract
CLIP has emerged as a powerful multimodal model capable of connecting images and text through joint embeddings, but to what extent does it "see" the same way humans do - especially when interpreting artworks? In this paper, we investigate CLIP's ability to extract high-level semantic and stylistic information from paintings, including both human-created and AI-generated imagery. We evaluate its perception across multiple dimensions: content, scene understanding, artistic style, historical period, and the presence of visual deformations or artifacts. By designing targeted probing tasks and comparing CLIP's responses to human annotations and expert benchmarks, we explore its alignment with human perceptual and contextual understanding. Our findings reveal both strengths and limitations in CLIP's visual representations, particularly in relation to aesthetic cues and artistic intent. We further discuss the implications of these insights for using CLIP as a guidance mechanism during generative processes, such as style transfer or prompt-based image synthesis. Our work highlights the need for deeper interpretability in multimodal systems, especially when applied to creative domains where nuance and subjectivity play a central role.
Abstract (translated)
CLIP作为一种能够通过联合嵌入连接图像和文本的强大多模态模型已经出现，但其“视觉”能力在多大程度上与人类相似——特别是在解读艺术品时？本文探讨了CLIP从绘画中提取高层次语义和风格信息的能力，包括由人创作的和AI生成的图像。我们通过多个维度评估其感知力：内容、场景理解、艺术风格、历史时期以及视觉变形或瑕疵的存在。通过设计有针对性的探测任务，并将CLIP的响应与人类注释和专家基准进行比较，我们探讨了它在人类知觉和语境理解方面的对齐情况。我们的研究结果揭示了CLIP在视觉表示上的强项和局限性，特别是在美学线索和艺术意图方面。此外，我们还讨论了这些见解对于使用CLIP作为生成过程（如风格转换或基于提示的图像合成）中的指导机制的影响。我们的工作强调了多模态系统在应用于需要微妙性和主观性的创意领域时对更深解读的需求。
URL
https://arxiv.org/abs/2505.05229
PDF
-
RLMiniStyler: Light-weight RL Style Agent for Arbitrary Sequential Neural Style Generation
2025-05-07 13:57:42Jing Hu, Chengming Feng, Shu Hu, Ming-Ching Chang, Xin Li, Xi Wu, Xin WangAbstract
Arbitrary style transfer aims to apply the style of any given artistic image to another content image. Still, existing deep learning-based methods often require significant computational costs to generate diverse stylized results. Motivated by this, we propose a novel reinforcement learning-based framework for arbitrary style transfer RLMiniStyler. This framework leverages a unified reinforcement learning policy to iteratively guide the style transfer process by exploring and exploiting stylization feedback, generating smooth sequences of stylized results while achieving model lightweight. Furthermore, we introduce an uncertainty-aware multi-task learning strategy that automatically adjusts loss weights to adapt to the content and style balance requirements at different training stages, thereby accelerating model convergence. Through a series of experiments across image various resolutions, we have validated the advantages of RLMiniStyler over other state-of-the-art methods in generating high-quality, diverse artistic image sequences at a lower cost. Codes are available at this https URL.
Abstract (translated)
任意风格转换的目标是将给定的艺术图像的风格应用到另一张内容图上。然而，现有的基于深度学习的方法通常需要较大的计算成本来生成多样化的风格化结果。鉴于此，我们提出了一种新颖的强化学习框架RLMiniStyler用于任意风格转移。该框架利用统一的强化学习策略迭代地引导风格转换过程，通过探索和利用风格化反馈，从而在实现模型轻量化的同时生成平滑连续的风格化结果序列。 此外，我们引入了基于不确定性的多任务学习策略，这种策略可以自动调整损失权重以适应不同训练阶段的内容与风格平衡需求，从而加速模型收敛。通过一系列跨图像分辨率的实验，我们在RLMiniStyler与其他最新方法之间验证了其在低成本下生成高质量、多样化艺术图序列的优势。 代码可以在提供的链接处获取：[此URL]（请将括号内的内容替换为实际链接）。
URL
https://arxiv.org/abs/2505.04424
PDF
-
Steering Large Language Models with Register Analysis for Arbitrary Style Transfer
2025-05-01 17:39:02Xinchen Yang, Marine CarpuatAbstract
Large Language Models (LLMs) have demonstrated strong capabilities in rewriting text across various styles. However, effectively leveraging this ability for example-based arbitrary style transfer, where an input text is rewritten to match the style of a given exemplar, remains an open challenge. A key question is how to describe the style of the exemplar to guide LLMs toward high-quality rewrites. In this work, we propose a prompting method based on register analysis to guide LLMs to perform this task. Empirical evaluations across multiple style transfer tasks show that our prompting approach enhances style transfer strength while preserving meaning more effectively than existing prompting strategies.
Abstract (translated)
大型语言模型（LLMs）在跨各种风格改写文本方面表现出强大的能力。然而，如何有效利用这种能力进行基于示例的任意风格转换仍是一个开放性挑战，尤其是在将输入文本改写为与给定示例相匹配的风格时。关键问题是应如何描述示例的风格以引导LLMs生成高质量的重写内容。在本研究中，我们提出了一种基于语域分析的提示方法来指导LLMs完成这一任务。通过多项风格转换任务的实际评估表明，我们的提示策略相比现有的提示策略能够更有效地增强风格转换能力同时更好地保留原意。
URL
https://arxiv.org/abs/2505.00679
PDF
-
Towards Generalized and Training-Free Text-Guided Semantic Manipulation
2025-04-24 05:54:56Yu Hong, Xiao Cai, Pengpeng Zeng, Shuai Zhang, Jingkuan Song, Lianli Gao, Heng Tao ShenAbstract
Text-guided semantic manipulation refers to semantically editing an image generated from a source prompt to match a target prompt, enabling the desired semantic changes (e.g., addition, removal, and style transfer) while preserving irrelevant contents. With the powerful generative capabilities of the diffusion model, the task has shown the potential to generate high-fidelity visual content. Nevertheless, existing methods either typically require time-consuming fine-tuning (inefficient), fail to accomplish multiple semantic manipulations (poorly extensible), and/or lack support for different modality tasks (limited generalizability). Upon further investigation, we find that the geometric properties of noises in the diffusion model are strongly correlated with the semantic changes. Motivated by this, we propose a novel $\textit{GTF}$ for text-guided semantic manipulation, which has the following attractive capabilities: 1) $\textbf{Generalized}$: our $\textit{GTF}$ supports multiple semantic manipulations (e.g., addition, removal, and style transfer) and can be seamlessly integrated into all diffusion-based methods (i.e., Plug-and-play) across different modalities (i.e., modality-agnostic); and 2) $\textbf{Training-free}$: $\textit{GTF}$ produces high-fidelity results via simply controlling the geometric relationship between noises without tuning or optimization. Our extensive experiments demonstrate the efficacy of our approach, highlighting its potential to advance the state-of-the-art in semantics manipulation.
Abstract (translated)
文本引导的语义编辑是指在基于源提示生成图像后，通过对该图像进行语义上的修改使其与目标描述相匹配的过程。这个过程可以实现诸如添加、移除或风格转换等所需的语义变化，同时保持无关内容不变。借助扩散模型强大的生成能力，这项任务已经显示出能够生成高保真视觉内容的潜力。然而，现有的方法要么需要耗时的微调（效率低下），无法完成多样的语义编辑操作（扩展性差），或者不支持不同模态的任务（泛化能力弱）。进一步研究发现，扩散模型中噪声的几何属性与语义变化之间有很强的相关性。受此启发，我们提出了一种新颖的方法——$\textit{GTF}$ (Generalized Training-free Framework)，用于文本引导的语义编辑，该方法具备以下优点： 1. **通用化**：我们的 $\textit{GTF}$ 支持多种语义操作（如添加、移除和风格转换），并且可以无缝地集成到所有基于扩散模型的方法中，在不同模态下均能使用（即插即用）。 2. **无需训练**：$\textit{GTF}$ 通过简单控制噪声之间的几何关系即可生成高保真的结果，而不需要任何微调或优化步骤。 我们的大量实验展示了这种方法的有效性，并突显了其在语义编辑领域中的领先地位和应用潜力。
URL
https://arxiv.org/abs/2504.17269
PDF
-
StyleMe3D: Stylization with Disentangled Priors by Multiple Encoders on 3D Gaussians
2025-04-21 17:59:55Cailin Zhuang, Yaoqi Hu, Xuanyang Zhang, Wei Cheng, Jiacheng Bao, Shengqi Liu, Yiying Yang, Xianfang Zeng, Gang Yu, Ming LiAbstract
3D Gaussian Splatting (3DGS) excels in photorealistic scene reconstruction but struggles with stylized scenarios (e.g., cartoons, games) due to fragmented textures, semantic misalignment, and limited adaptability to abstract aesthetics. We propose StyleMe3D, a holistic framework for 3D GS style transfer that integrates multi-modal style conditioning, multi-level semantic alignment, and perceptual quality enhancement. Our key insights include: (1) optimizing only RGB attributes preserves geometric integrity during stylization; (2) disentangling low-, medium-, and high-level semantics is critical for coherent style transfer; (3) scalability across isolated objects and complex scenes is essential for practical deployment. StyleMe3D introduces four novel components: Dynamic Style Score Distillation (DSSD), leveraging Stable Diffusion's latent space for semantic alignment; Contrastive Style Descriptor (CSD) for localized, content-aware texture transfer; Simultaneously Optimized Scale (SOS) to decouple style details and structural coherence; and 3D Gaussian Quality Assessment (3DG-QA), a differentiable aesthetic prior trained on human-rated data to suppress artifacts and enhance visual harmony. Evaluated on NeRF synthetic dataset (objects) and tandt db (scenes) datasets, StyleMe3D outperforms state-of-the-art methods in preserving geometric details (e.g., carvings on sculptures) and ensuring stylistic consistency across scenes (e.g., coherent lighting in landscapes), while maintaining real-time rendering. This work bridges photorealistic 3D GS and artistic stylization, unlocking applications in gaming, virtual worlds, and digital art.
Abstract (translated)
3D高斯点云（3DGS）在逼真的场景重建方面表现出色，但在处理风格化场景（例如卡通、游戏等）时遇到了挑战。由于纹理碎片化、语义不匹配和对抽象美学适应性有限的原因，它在这方面表现不佳。为此，我们提出了StyleMe3D——一个用于3D GS风格转换的综合性框架，该框架集成了多模态风格条件、多层次语义对齐以及感知质量增强。我们的关键见解包括： 1. 仅优化RGB属性可以在风格化过程中保持几何完整性。 2. 解耦低级、中级和高级语义对于连贯的风格转移至关重要。 3. 对孤立对象和复杂场景的扩展性是实际部署中的必要条件。 StyleMe3D引入了四个新颖组件：动态风格分数蒸馏（Dynamic Style Score Distillation，DSSD），利用Stable Diffusion的潜在空间进行语义对齐；对比风格描述符（Contrastive Style Descriptor，CSD），用于局部化、内容感知的纹理转移；同时优化尺度（Simultaneously Optimized Scale，SOS），以解耦样式细节和结构一致性；以及3D高斯质量评估（3DG-QA），这是一种基于人类评分数据训练而成的可微美学术前项，能够抑制伪影并增强视觉和谐。 在NeRF合成数据集（对象）和tandt db（场景）数据集中进行评估后，StyleMe3D在保留几何细节（例如雕塑上的雕刻）以及确保跨场景风格一致性（例如景观中的连贯照明）方面优于现有最先进的方法，并且能够保持实时渲染。这项工作将逼真的3D GS与艺术风格化结合在一起，解锁了游戏、虚拟世界和数字艺术领域的应用。
URL
https://arxiv.org/abs/2504.15281
PDF
-
IXGS-Intraoperative 3D Reconstruction from Sparse, Arbitrarily Posed Real X-rays
2025-04-20 18:28:13Sascha Jecklin, Aidana Massalimova, Ruyi Zha, Lilian Calvet, Christoph J. Laux, Mazda Farshad, Philipp F\"urnstahlAbstract
Spine surgery is a high-risk intervention demanding precise execution, often supported by image-based navigation systems. Recently, supervised learning approaches have gained attention for reconstructing 3D spinal anatomy from sparse fluoroscopic data, significantly reducing reliance on radiation-intensive 3D imaging systems. However, these methods typically require large amounts of annotated training data and may struggle to generalize across varying patient anatomies or imaging conditions. Instance-learning approaches like Gaussian splatting could offer an alternative by avoiding extensive annotation requirements. While Gaussian splatting has shown promise for novel view synthesis, its application to sparse, arbitrarily posed real intraoperative X-rays has remained largely unexplored. This work addresses this limitation by extending the $R^2$-Gaussian splatting framework to reconstruct anatomically consistent 3D volumes under these challenging conditions. We introduce an anatomy-guided radiographic standardization step using style transfer, improving visual consistency across views, and enhancing reconstruction quality. Notably, our framework requires no pretraining, making it inherently adaptable to new patients and anatomies. We evaluated our approach using an ex-vivo dataset. Expert surgical evaluation confirmed the clinical utility of the 3D reconstructions for navigation, especially when using 20 to 30 views, and highlighted the standardization's benefit for anatomical clarity. Benchmarking via quantitative 2D metrics (PSNR/SSIM) confirmed performance trade-offs compared to idealized settings, but also validated the improvement gained from standardization over raw inputs. This work demonstrates the feasibility of instance-based volumetric reconstruction from arbitrary sparse-view X-rays, advancing intraoperative 3D imaging for surgical navigation.
Abstract (translated)
脊柱手术是一项高风险的干预措施，要求精确执行，并且通常需要图像导航系统的支持。最近，监督学习方法因能从稀疏的荧光透视数据中重建三维脊柱解剖结构而受到关注，这大大减少了对放射强度高的3D成像系统的依赖。然而，这些方法通常需要大量的注释训练数据，并可能难以在不同的患者解剖结构或成像条件下进行泛化。实例学习方法（如高斯点阵）通过避免大量标注要求提供了一种替代方案。尽管高斯点阵法在新颖视图合成方面显示出潜力，但将其应用于稀疏、任意角度的实时手术X射线的应用尚属鲜有探索。 本研究旨在解决这一限制，扩展了$R^2$-高斯点阵框架以在这种具有挑战性的条件下重建解剖一致的三维体积。我们引入了一步基于解剖引导的放射学标准化步骤，通过样式转换改善跨视图的一致性并提高重建质量。值得注意的是，我们的框架不需要预训练，从而使其能够适应新的患者和解剖结构。 我们在一个离体数据集上评估了这种方法，并由专家手术评价确认3D重建在导航中的临床实用性，尤其是在使用20到30个视角时，特别强调标准化对解剖清晰度的益处。通过定量二维指标（PSNR/SSIM）进行基准测试证实，在理想设置中存在性能折衷，但也验证了从原始输入到标准化改进的有效性。 这项工作展示了基于实例的体积重建技术从任意稀疏视图X射线获取中的可行性，并推动了手术导航中的实时3D成像技术。
URL
https://arxiv.org/abs/2504.14699
PDF
-
ICAS: IP Adapter and ControlNet-based Attention Structure for Multi-Subject Style Transfer Optimization
2025-04-17 10:48:11Fuwei LiuAbstract
Generating multi-subject stylized images remains a significant challenge due to the ambiguity in defining style attributes (e.g., color, texture, atmosphere, and structure) and the difficulty in consistently applying them across multiple subjects. Although recent diffusion-based text-to-image models have achieved remarkable progress, existing methods typically rely on computationally expensive inversion procedures or large-scale stylized datasets. Moreover, these methods often struggle with maintaining multi-subject semantic fidelity and are limited by high inference costs. To address these limitations, we propose ICAS (IP-Adapter and ControlNet-based Attention Structure), a novel framework for efficient and controllable multi-subject style transfer. Instead of full-model tuning, ICAS adaptively fine-tunes only the content injection branch of a pre-trained diffusion model, thereby preserving identity-specific semantics while enhancing style controllability. By combining IP-Adapter for adaptive style injection with ControlNet for structural conditioning, our framework ensures faithful global layout preservation alongside accurate local style synthesis. Furthermore, ICAS introduces a cyclic multi-subject content embedding mechanism, which enables effective style transfer under limited-data settings without the need for extensive stylized corpora. Extensive experiments show that ICAS achieves superior performance in structure preservation, style consistency, and inference efficiency, establishing a new paradigm for multi-subject style transfer in real-world applications.
Abstract (translated)
生成多主体风格化图像仍然面临重大挑战，主要是由于定义风格属性（如颜色、纹理、氛围和结构）的模糊性以及在多个主体之间一致应用这些属性的难度。尽管最近基于扩散模型的文本到图像方法取得了显著进展，现有的方法通常依赖于计算成本高昂的逆向过程或大规模风格化数据集。此外，这些方法往往难以保持多主体语义保真度，并且受到高推断成本的限制。 为了克服这些局限性，我们提出了ICAS（IP-Adapter 和 ControlNet 基础注意力结构），这是一个新型框架，旨在实现高效可控的多主体风格转换。不同于全模型微调，ICAS 仅对预训练扩散模型的内容注入分支进行自适应微调，从而在保留特定身份语义的同时增强风格控制性。通过结合IP-Adapter 进行自适应风格注入与ControlNet 的结构条件设定，我们的框架确保了忠实的全局布局保存及精确的地方风格合成。 此外，ICAS 引入了一种循环多主体内容嵌入机制，在数据有限的情况下实现了有效的风格转换，无需依赖大规模风格化语料库。广泛的实验表明，ICAS 在结构保留、风格一致性和推断效率方面取得了卓越性能，为现实世界应用中的多主题风格转移开创了新的范式。
URL
https://arxiv.org/abs/2504.13224
PDF
-
IlluSign: Illustrating Sign Language Videos by Leveraging the Attention Mechanism
2025-04-15 02:53:32Janna Bruner, Amit Moryossef, Lior WolfAbstract
Sign languages are dynamic visual languages that involve hand gestures, in combination with non manual elements such as facial expressions. While video recordings of sign language are commonly used for education and documentation, the dynamic nature of signs can make it challenging to study them in detail, especially for new learners and educators. This work aims to convert sign language video footage into static illustrations, which serve as an additional educational resource to complement video content. This process is usually done by an artist, and is therefore quite costly. We propose a method that illustrates sign language videos by leveraging generative models' ability to understand both the semantic and geometric aspects of images. Our approach focuses on transferring a sketch like illustration style to video footage of sign language, combining the start and end frames of a sign into a single illustration, and using arrows to highlight the hand's direction and motion. While many style transfer methods address domain adaptation at varying levels of abstraction, applying a sketch like style to sign languages, especially for hand gestures and facial expressions, poses a significant challenge. To tackle this, we intervene in the denoising process of a diffusion model, injecting style as keys and values into high resolution attention layers, and fusing geometric information from the image and edges as queries. For the final illustration, we use the attention mechanism to combine the attention weights from both the start and end illustrations, resulting in a soft combination. Our method offers a cost effective solution for generating sign language illustrations at inference time, addressing the lack of such resources in educational materials.
Abstract (translated)
手语是一种动态的视觉语言，通过手势结合面部表情等非手动元素来表达。虽然视频录制的手语在教育和记录中被广泛使用，但由于其动态特性，在细节上进行研究具有挑战性，尤其是对于新手学习者和教师来说更为困难。本项目旨在将手语视频片段转化为静态插图，作为补充视频内容的教育资源。这一过程通常由艺术家完成，因此成本较高。我们提出了一种方法，利用生成模型理解图像的语义和几何方面的能力来绘制手语视频。我们的方法重点在于将类似素描的风格转移到手语视频上，并结合手势开始帧和结束帧以形成单一插图，同时使用箭头突出双手的方向和运动。 虽然许多风格迁移的方法在不同程度上解决了领域适应的问题，但对手势语言特别是对于手部动作和面部表情进行素描化处理仍然具有挑战性。为此，我们对扩散模型的去噪过程进行了干预，在高分辨率注意力层中注入样式作为键值，并融合图像和边缘中的几何信息作为查询。最后，通过注意机制结合开始帧和结束帧插图的关注权重来产生最终的插图，形成柔和组合。 我们的方法提供了一种成本效益高的解决方案，可以在推断时生成手语插图，弥补了教育材料中此类资源的不足。
URL
https://arxiv.org/abs/2504.10822
PDF
