Style_Transfer
-
Data Augmentation Through Random Style Replacement
2025-04-14 16:46:32Qikai Yang, Cheng Ji, Huaiying Luo, Panfeng Li, Zhicheng DingAbstract
In this paper, we introduce a novel data augmentation technique that combines the advantages of style augmentation and random erasing by selectively replacing image subregions with style-transferred patches. Our approach first applies a random style transfer to training images, then randomly substitutes selected areas of these images with patches derived from the style-transferred versions. This method is able to seamlessly accommodate a wide range of existing style transfer algorithms and can be readily integrated into diverse data augmentation pipelines. By incorporating our strategy, the training process becomes more robust and less prone to overfitting. Comparative experiments demonstrate that, relative to previous style augmentation methods, our technique achieves superior performance and faster convergence.
Abstract (translated)
在本文中，我们介绍了一种新颖的数据增强技术，该技术结合了样式增强和随机擦除的优点，通过选择性地用经过风格转换的补丁替换图像子区域来实现。我们的方法首先对训练图像应用随机样式转换，然后从这些经过样式转换的版本中选取部分区域作为补丁，并将原始图像中的选定区域随机替换成这些补丁。这种方法能够无缝兼容各种现有的风格转移算法，并且可以轻松集成到多种数据增强流程中。通过采用我们的策略，训练过程变得更加稳健，减少了过拟合的风险。比较实验表明，相对于先前的样式增强方法，我们的技术在性能上更胜一筹，并且收敛速度更快。
URL
https://arxiv.org/abs/2504.10563
PDF
-
Trade-offs in Privacy-Preserving Eye Tracking through Iris Obfuscation: A Benchmarking Study
2025-04-14 14:29:38Mengdi Wang, Efe Bozkir, Enkelejda KasneciAbstract
Recent developments in hardware, computer graphics, and AI may soon enable AR/VR head-mounted displays (HMDs) to become everyday devices like smartphones and tablets. Eye trackers within HMDs provide a special opportunity for such setups as it is possible to facilitate gaze-based research and interaction. However, estimating users' gaze information often requires raw eye images and videos that contain iris textures, which are considered a gold standard biometric for user authentication, and this raises privacy concerns. Previous research in the eye-tracking community focused on obfuscating iris textures while keeping utility tasks such as gaze estimation accurate. Despite these attempts, there is no comprehensive benchmark that evaluates state-of-the-art approaches. Considering all, in this paper, we benchmark blurring, noising, downsampling, rubber sheet model, and iris style transfer to obfuscate user identity, and compare their impact on image quality, privacy, utility, and risk of imposter attack on two datasets. We use eye segmentation and gaze estimation as utility tasks, and reduction in iris recognition accuracy as a measure of privacy protection, and false acceptance rate to estimate risk of attack. Our experiments show that canonical image processing methods like blurring and noising cause a marginal impact on deep learning-based tasks. While downsampling, rubber sheet model, and iris style transfer are effective in hiding user identifiers, iris style transfer, with higher computation cost, outperforms others in both utility tasks, and is more resilient against spoof attacks. Our analyses indicate that there is no universal optimal approach to balance privacy, utility, and computation burden. Therefore, we recommend practitioners consider the strengths and weaknesses of each approach, and possible combinations of those to reach an optimal privacy-utility trade-off.
Abstract (translated)
近期在硬件、计算机图形和人工智能方面的进展可能会使增强现实/虚拟现实头戴式显示器（HMD）成为像智能手机和平板电脑一样日常使用的设备。HMD 中的眼动追踪器为这种设置提供了特殊的机会，因为它可以促进基于注视的研究和交互。然而，估计用户的目光信息通常需要包含虹膜纹理的原始眼图像和视频，而这些被视作用于身份认证的标准生物特征数据，这引发了隐私方面的担忧。此前，在眼动追踪领域的研究主要集中在模糊化虹膜纹理的同时保持诸如视线估计等实用任务的准确性。尽管有这些尝试，但尚无全面评估前沿方法的基准测试。 考虑到所有因素，本文对模糊、噪声处理、降采样、橡胶片模型和虹膜风格转换这五种技术进行基准测试，以混淆用户身份，并比较它们在两个数据集上对图像质量、隐私性、实用性和冒充攻击风险的影响。我们使用眼睛分割和注视估计作为实用性任务，将虹膜识别准确性的降低视为隐私保护的衡量标准，并用错误接受率来估算攻击的风险。 我们的实验表明，传统的图像处理方法如模糊和添加噪声对于基于深度学习的任务影响甚微。然而，降采样、橡胶片模型以及虹膜风格转换在隐藏用户标识符方面都表现出色，其中虹膜风格转换以更高的计算成本为代价，在实用性任务中表现优于其他技术，并且更能抵御冒充攻击。 我们的分析表明，没有一种普遍适用的方法可以在隐私性、实用性和计算负担之间达到平衡。因此，我们建议实践者应考虑每种方法的优点和缺点，以及这些方法可能的组合方式，以实现最佳的隐私与实用性权衡。
URL
https://arxiv.org/abs/2504.10267
PDF
-
Muon-Accelerated Attention Distillation for Real-Time Edge Synthesis via Optimized Latent Diffusion
2025-04-11 11:27:29Weiye Chen, Qingen Zhu, Qian LongAbstract
Recent advances in visual synthesis have leveraged diffusion models and attention mechanisms to achieve high-fidelity artistic style transfer and photorealistic text-to-image generation. However, real-time deployment on edge devices remains challenging due to computational and memory constraints. We propose Muon-AD, a co-designed framework that integrates the Muon optimizer with attention distillation for real-time edge synthesis. By eliminating gradient conflicts through orthogonal parameter updates and dynamic pruning, Muon-AD achieves 3.2 times faster convergence compared to Stable Diffusion-TensorRT, while maintaining synthesis quality (15% lower FID, 4% higher SSIM). Our framework reduces peak memory to 7GB on Jetson Orin and enables 24FPS real-time generation through mixed-precision quantization and curriculum learning. Extensive experiments on COCO-Stuff and ImageNet-Texture demonstrate Muon-AD's Pareto-optimal efficiency-quality trade-offs. Here, we show a 65% reduction in communication overhead during distributed training and real-time 10s/image generation on edge GPUs. These advancements pave the way for democratizing high-quality visual synthesis in resource-constrained environments.
Abstract (translated)
最近在视觉合成领域取得的进展利用了扩散模型和注意力机制，实现了高质量的艺术风格转换以及逼真的文本到图像生成。然而，由于计算资源和内存限制，在边缘设备上实现实时部署仍然面临挑战。我们提出了一种名为Muon-AD的框架，该框架结合了Muon优化器与注意力蒸馏技术，旨在为边缘设备上的实时合成提供解决方案。 通过正交参数更新和动态修剪消除梯度冲突，Muon-AD实现了比Stable Diffusion-TensorRT快3.2倍的收敛速度，同时保持了合成质量（FID降低了15%，SSIM提高了4%）。我们的框架将Jetson Orin上的峰值内存减少到7GB，并通过混合精度量化和课程学习实现实时生成速率高达24FPS。在COCO-Stuff和ImageNet-Texture数据集上进行的广泛实验表明，Muon-AD在效率与质量之间实现了Pareto最优权衡。 此外，我们的方法展示了分布式训练期间通信开销减少了65%，并且能够在边缘GPU上实现实时每10秒生成一张图像。这些改进为在资源受限环境中实现高质量视觉合成铺平了道路。
URL
https://arxiv.org/abs/2504.08451
PDF
-
DreamFuse: Adaptive Image Fusion with Diffusion Transformer
2025-04-11 06:49:33Junjia Huang, Pengxiang Yan, Jiyang Liu, Jie Wu, Zhao Wang, Yitong Wang, Liang Lin, Guanbin LiAbstract
Image fusion seeks to seamlessly integrate foreground objects with background scenes, producing realistic and harmonious fused images. Unlike existing methods that directly insert objects into the background, adaptive and interactive fusion remains a challenging yet appealing task. It requires the foreground to adjust or interact with the background context, enabling more coherent integration. To address this, we propose an iterative human-in-the-loop data generation pipeline, which leverages limited initial data with diverse textual prompts to generate fusion datasets across various scenarios and interactions, including placement, holding, wearing, and style transfer. Building on this, we introduce DreamFuse, a novel approach based on the Diffusion Transformer (DiT) model, to generate consistent and harmonious fused images with both foreground and background information. DreamFuse employs a Positional Affine mechanism to inject the size and position of the foreground into the background, enabling effective foreground-background interaction through shared attention. Furthermore, we apply Localized Direct Preference Optimization guided by human feedback to refine DreamFuse, enhancing background consistency and foreground harmony. DreamFuse achieves harmonious fusion while generalizing to text-driven attribute editing of the fused results. Experimental results demonstrate that our method outperforms state-of-the-art approaches across multiple metrics.
Abstract (translated)
图像融合的目标是无缝地将前景对象与背景场景结合，生成逼真且和谐的合成图像。不同于现有的直接将对象插入到背景中的方法，适应性和交互式的融合仍然是一项具有挑战性但吸引人的任务。这一过程要求前景能够根据背景环境进行调整或互动，从而实现更连贯的整合。为此，我们提出了一种迭代的人机协作数据生成流水线，该流水线利用有限的初始数据和多样化的文本提示来生成涵盖不同场景和交互方式（如放置、手持、穿戴及风格转换）的融合数据集。在此基础上，我们引入了DreamFuse这一基于扩散变换器(DiT)模型的新方法，它能够结合前景与背景信息，生成一致且和谐的合成图像。DreamFuse采用位置归一化机制将前景对象的位置和大小注入到背景中，通过共享注意力实现有效的前、后景互动。此外，我们还应用了由人类反馈引导的局部直接偏好优化技术来改进DreamFuse，增强其在背景一致性及前景和谐性方面的表现。DreamFuse不仅实现了无缝融合，还能根据文本驱动的属性编辑需求进行泛化。实验结果表明，在多个评价指标上我们的方法超越了现有的最先进技术。
URL
https://arxiv.org/abs/2504.08291
PDF
-
StyleRec: A Benchmark Dataset for Prompt Recovery in Writing Style Transformation
2025-04-06 06:02:28Shenyang Liu, Yang Gao, Shaoyan Zhai, Liqiang WangAbstract
Prompt Recovery, reconstructing prompts from the outputs of large language models (LLMs), has grown in importance as LLMs become ubiquitous. Most users access LLMs through APIs without internal model weights, relying only on outputs and logits, which complicates recovery. This paper explores a unique prompt recovery task focused on reconstructing prompts for style transfer and rephrasing, rather than typical question-answering. We introduce a dataset created with LLM assistance, ensuring quality through multiple techniques, and test methods like zero-shot, few-shot, jailbreak, chain-of-thought, fine-tuning, and a novel canonical-prompt fallback for poor-performing cases. Our results show that one-shot and fine-tuning yield the best outcomes but highlight flaws in traditional sentence similarity metrics for evaluating prompt recovery. Contributions include (1) a benchmark dataset, (2) comprehensive experiments on prompt recovery strategies, and (3) identification of limitations in current evaluation metrics, all of which advance general prompt recovery research, where the structure of the input prompt is unrestricted.
Abstract (translated)
**翻译** 提示恢复，即从大型语言模型（LLM）的输出中重构提示信息，在LLMs广泛使用的情况下变得日益重要。大多数用户通过API访问LLMs，并且没有内部模型权重的支持，仅依赖于输出和logits，这使得提示恢复变得更加复杂。本文探讨了一个独特的任务——专注于重建用于风格转换和重新表述的提示信息，而不是传统的问答场景。我们介绍了一套利用LLM辅助创建的数据集，并采用多种方法确保数据质量。此外，还测试了零样本、少量样本、破解（jailbreak）、链式思维（chain-of-thought）以及针对性能不佳情况的新颖规范性提示回退等方法。我们的研究结果表明，单样本和微调策略能取得最佳效果，但同时也指出了传统句子相似度评估指标在评价提示恢复方面的不足之处。本文的贡献包括：(1) 提供了一个基准数据集；(2) 对多种提示恢复策略进行了全面实验；以及 (3) 发现了当前评估指标存在的局限性。所有这些都进一步推进了一般性的未受限输入提示结构的提示恢复研究。
URL
https://arxiv.org/abs/2504.04373
PDF
-
F-ViTA: Foundation Model Guided Visible to Thermal Translation
2025-04-03 17:47:06Jay N. Paranjape, Celso de Melo, Vishal M. PatelAbstract
Thermal imaging is crucial for scene understanding, particularly in low-light and nighttime conditions. However, collecting large thermal datasets is costly and labor-intensive due to the specialized equipment required for infrared image capture. To address this challenge, researchers have explored visible-to-thermal image translation. Most existing methods rely on Generative Adversarial Networks (GANs) or Diffusion Models (DMs), treating the task as a style transfer problem. As a result, these approaches attempt to learn both the modality distribution shift and underlying physical principles from limited training data. In this paper, we propose F-ViTA, a novel approach that leverages the general world knowledge embedded in foundation models to guide the diffusion process for improved translation. Specifically, we condition an InstructPix2Pix Diffusion Model with zero-shot masks and labels from foundation models such as SAM and Grounded DINO. This allows the model to learn meaningful correlations between scene objects and their thermal signatures in infrared imagery. Extensive experiments on five public datasets demonstrate that F-ViTA outperforms state-of-the-art (SOTA) methods. Furthermore, our model generalizes well to out-of-distribution (OOD) scenarios and can generate Long-Wave Infrared (LWIR), Mid-Wave Infrared (MWIR), and Near-Infrared (NIR) translations from the same visible image. Code: this https URL.
Abstract (translated)
热成像是场景理解的关键，尤其是在低光和夜间条件下。然而，收集大规模的热数据集由于需要专门的红外图像捕捉设备而成本高昂且耗时长。为了解决这一挑战，研究人员探索了可见光到热成像的转换技术。大多数现有的方法依赖于生成对抗网络（GANs）或扩散模型（DMs），将任务视为风格迁移问题。因此，这些方法试图从有限的训练数据中学习模态分布变化和潜在的物理原理。 在本文中，我们提出了F-ViTA，这是一种新颖的方法，它利用基础模型中嵌入的一般世界知识来指导扩散过程以实现更优的转换效果。具体而言，我们将InstructPix2Pix扩散模型与来自SAM（Segment Anything Model）和Grounded DINO等基础模型的零样本掩码和标签相结合进行条件化处理。这样可以使模型学习场景中的物体与其在红外图像中的热特征之间的有意义的相关性。 我们在五个公共数据集上进行了广泛的实验，结果显示F-ViTA优于当前最先进（SOTA）的方法。此外，我们的模型能够在分布外（OOD）情景下表现出良好的泛化能力，并且可以从同一张可见光图像生成长波红外（LWIR）、中波红外（MWIR）和近红外（NIR）的转换结果。 代码地址：[这个链接](https://this https URL)
URL
https://arxiv.org/abs/2504.02801
PDF
-
Neural Style Transfer for Synthesising a Dataset of Ancient Egyptian Hieroglyphs
2025-04-02 22:30:45Lewis Matheson CreedAbstract
The limited availability of training data for low-resource languages makes applying machine learning techniques challenging. Ancient Egyptian is one such language with few resources. However, innovative applications of data augmentation methods, such as Neural Style Transfer, could overcome these barriers. This paper presents a novel method for generating datasets of ancient Egyptian hieroglyphs by applying NST to a digital typeface. Experimental results found that image classification models trained on NST-generated examples and photographs demonstrate equal performance and transferability to real unseen images of hieroglyphs.
Abstract (translated)
低资源语言训练数据的有限可用性使得应用机器学习技术变得具有挑战性。古埃及语就是这样一个资源匮乏的语言。然而，通过创新地使用数据增强方法（如神经风格迁移）可以克服这些障碍。本文提出了一种新颖的方法，利用神经风格迁移对数字字体进行处理来生成古埃及象形文字的数据集。实验结果表明，在使用NST生成的示例和照片训练的图像分类模型在识别真实未见过的象形文字图像时表现出同等的效果和可转移性。
URL
https://arxiv.org/abs/2504.02163
PDF
-
Semantix: An Energy Guided Sampler for Semantic Style Transfer
2025-03-28 11:34:30Huiang He, Minghui Hu, Chuanxia Zheng, Chaoyue Wang, Tat-Jen ChamAbstract
Recent advances in style and appearance transfer are impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspondence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, Semantic Style Transfer, which involves transferring style and appearance features from a reference image to a target visual content based on semantic correspondence. We subsequently propose a training-free method, Semantix an energy-guided sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, as a sampler, Semantix be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, Semantix utilizes a meticulously crafted energy function to guide the sampling process, including three key components: Style Feature Guidance, Spatial Feature Guidance and Semantic Distance as a regularisation term. Experimental results demonstrate that Semantix not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields. The project website is available at this https URL
Abstract (translated)
最近在风格和外观转移方面的进展令人印象深刻，但大多数方法要么单独处理全局风格或局部外观的转换，要么忽视了语义对应关系。此外，图像和视频任务通常被孤立地处理，很少关注将它们集成以进行视频转移的方法。为了解决这些限制，我们引入了一个新的任务：语义风格迁移（Semantic Style Transfer），它涉及根据语义对应关系从参考图像中传输样式和外观特征到目标视觉内容。 随后，我们提出了一种无需训练的方法——Semantix，这是一个能量引导采样器，专门为语义风格转移设计。该方法基于预训练扩散模型的语义理解能力同时指导样式和外观转换。此外，作为采样器，Semantix可以无缝应用于图像和视频模型，使语义风格迁移能够在各种视觉媒体中通用。 具体而言，在将参考图和上下文图或视频通过SDEs反演到噪声空间后，Semantix利用精心设计的能量函数来指导采样过程。该能量函数包括三个关键组成部分：样式特征引导、空间特征引导以及作为正则化项的语义距离。 实验结果表明，Semantix不仅能够有效地完成图像和视频之间的语义风格迁移任务，并且在两个领域都超越了现有的最先进解决方案。该项目网站可在此处访问（请将“this https URL”替换为实际网址）。
URL
https://arxiv.org/abs/2503.22344
PDF
-
ABC-GS: Alignment-Based Controllable Style Transfer for 3D Gaussian Splatting
2025-03-28 08:07:57Wenjie Liu, Zhongliang Liu, Xiaoyan Yang, Man Sha, Yang LiAbstract
3D scene stylization approaches based on Neural Radiance Fields (NeRF) achieve promising results by optimizing with Nearest Neighbor Feature Matching (NNFM) loss. However, NNFM loss does not consider global style information. In addition, the implicit representation of NeRF limits their fine-grained control over the resulting scenes. In this paper, we introduce ABC-GS, a novel framework based on 3D Gaussian Splatting to achieve high-quality 3D style transfer. To this end, a controllable matching stage is designed to achieve precise alignment between scene content and style features through segmentation masks. Moreover, a style transfer loss function based on feature alignment is proposed to ensure that the outcomes of style transfer accurately reflect the global style of the reference image. Furthermore, the original geometric information of the scene is preserved with the depth loss and Gaussian regularization terms. Extensive experiments show that our ABC-GS provides controllability of style transfer and achieves stylization results that are more faithfully aligned with the global style of the chosen artistic reference. Our homepage is available at this https URL.
Abstract (translated)
基于神经辐射场（NeRF）的3D场景风格化方法通过最近邻特征匹配（NNFM）损失优化取得了令人鼓舞的结果。然而，NNFM损失没有考虑全局样式信息。此外，NeRF 的隐式表示限制了对结果场景的细粒度控制。在本文中，我们介绍了ABC-GS，这是一个基于3D高斯点绘的新框架，旨在实现高质量的3D风格转换。为此，设计了一个可控匹配阶段，通过分割掩码实现场景内容和样式特征之间的精确对齐。此外，提出了一种基于特征对齐的风格迁移损失函数，以确保风格迁移的结果准确反映参考图像的整体样式。另外，使用深度损失和高斯正则化项保留了原始场景的几何信息。广泛的实验表明，我们的ABC-GS提供了风格转换的可控性，并且实现了更加忠实于所选艺术参考的整体样式的风格化结果。更多关于ABC-GS的信息可以在[这里](https://example.com/)找到。
URL
https://arxiv.org/abs/2503.22218
PDF
-
Improving the generalization of deep learning models in the segmentation of mammography images
2025-03-28 00:11:00Jan Hurtado, Joao P. Maia, Cesar A. Sierra-Franco, Alberto RaposoAbstract
Mammography stands as the main screening method for detecting breast cancer early, enhancing treatment success rates. The segmentation of landmark structures in mammography images can aid the medical assessment in the evaluation of cancer risk and the image acquisition adequacy. We introduce a series of data-centric strategies aimed at enriching the training data for deep learning-based segmentation of landmark structures. Our approach involves augmenting the training samples through annotation-guided image intensity manipulation and style transfer to achieve better generalization than standard training procedures. These augmentations are applied in a balanced manner to ensure the model learns to process a diverse range of images generated by different vendor equipments while retaining its efficacy on the original data. We present extensive numerical and visual results that demonstrate the superior generalization capabilities of our methods when compared to the standard training. For this evaluation, we consider a large dataset that includes mammography images generated by different vendor equipments. Further, we present complementary results that show both the strengths and limitations of our methods across various scenarios. The accuracy and robustness demonstrated in the experiments suggest that our method is well-suited for integration into clinical practice.
Abstract (translated)
乳房X光摄影是早期检测乳腺癌的主要筛查方法，可以提高治疗成功率。在乳房X光图像中对标志性结构进行分割有助于医疗评估，在癌症风险评估和影像获取充分性方面发挥重要作用。我们提出了一系列以数据为中心的策略，旨在丰富基于深度学习的标志性结构分割模型的训练数据。我们的方法通过指导注释引导下的图像强度调整和风格转换来扩充训练样本，从而实现比标准训练程序更好的泛化能力。这些增强措施是以平衡的方式应用的，确保模型能够处理由不同供应商设备生成的不同类型的图像，同时在原始数据上保持其有效性。我们展示了大量的数值和视觉结果，证明了我们的方法与标准训练相比具有更优越的泛化能力。为此评估，我们考虑了一个包含来自不同供应商设备产生的乳房X光摄影图像的大规模数据集。此外，还呈现了补充结果，展示我们在各种场景下所提方法的优点和局限性。实验中表现出的准确性和鲁棒性表明我们的方法非常适合在临床实践中应用。
URL
https://arxiv.org/abs/2503.22052
PDF
-
Tune It Up: Music Genre Transfer and Prediction
2025-03-27 21:55:57Fidan Samet, Oguz Bakir, Adnan FidanAbstract
Deep generative models have been used in style transfer tasks for images. In this study, we adapt and improve CycleGAN model to perform music style transfer on Jazz and Classic genres. By doing so, we aim to easily generate new songs, cover music to different music genres and reduce the arrangements needed in those processes. We train and use music genre classifier to assess the performance of the transfer models. To that end, we obtain 87.7% accuracy with Multi-layer Perceptron algorithm. To improve our style transfer baseline, we add auxiliary discriminators and triplet loss to our model. According to our experiments, we obtain the best accuracies as 69.4% in Jazz to Classic task and 39.3% in Classic to Jazz task with our developed genre classifier. We also run a subjective experiment and results of it show that the overall performance of our transfer model is good and it manages to conserve melody of inputs on the transferred outputs. Our code is available at this https URL fidansamet/tune-it-up
Abstract (translated)
深度生成模型已被应用于图像风格转换任务中。在本研究中，我们对CycleGAN模型进行了适应和改进，以实现爵士乐和古典音乐之间的风格转移。通过这种方式，我们的目标是轻松地生成新歌曲、将音乐改编为不同的音乐流派，并减少这些过程中的编曲需求。为了评估风格迁移模型的性能，我们训练并使用了音乐流派分类器。基于此，我们利用多层感知机算法达到了87.7%的准确率。 为了提高我们的风格转移基线，我们在模型中添加了辅助鉴别器和三元组损失函数。根据我们的实验结果，在爵士乐到古典音乐的任务中，我们获得了69.4%的最佳准确性；而在古典音乐到爵士乐的任务中，则是39.3%的最高准确率，这些成绩都是通过使用我们开发的流派分类器得到的。 此外，我们也进行了一项主观实验。该实验的结果表明，我们的转移模型的整体性能良好，并且能够保持输入旋律在转换输出中的完整性。 我们的代码可以在以下链接中找到：https://github.com/fidansamet/tune-it-up 请注意，上述URL是示例性的，在实际使用时需要替换为正确的GitHub地址或相应的开源平台地址。
URL
https://arxiv.org/abs/2503.22008
PDF
-
Zero-Shot Visual Concept Blending Without Text Guidance
2025-03-27 08:56:33Hiroya Makino, Takahiro Yamaguchi, Hiroyuki SakaiAbstract
We propose a novel, zero-shot image generation technique called "Visual Concept Blending" that provides fine-grained control over which features from multiple reference images are transferred to a source image. If only a single reference image is available, it is difficult to isolate which specific elements should be transferred. However, using multiple reference images, the proposed approach distinguishes between common and unique features by selectively incorporating them into a generated output. By operating within a partially disentangled Contrastive Language-Image Pre-training (CLIP) embedding space (from IP-Adapter), our method enables the flexible transfer of texture, shape, motion, style, and more abstract conceptual transformations without requiring additional training or text prompts. We demonstrate its effectiveness across a diverse range of tasks, including style transfer, form metamorphosis, and conceptual transformations, showing how subtle or abstract attributes (e.g., brushstroke style, aerodynamic lines, and dynamism) can be seamlessly combined into a new image. In a user study, participants accurately recognized which features were intended to be transferred. Its simplicity, flexibility, and high-level control make Visual Concept Blending valuable for creative fields such as art, design, and content creation, where combining specific visual qualities from multiple inspirations is crucial.
Abstract (translated)
我们提出了一种新颖的零样本图像生成技术，名为“视觉概念融合”，该技术能够精细控制从多张参考图像中的哪些特征转移到源图像。当仅有一张参考图像可用时，很难确定应转移的具体元素。然而，通过使用多张参考图像，所提出的这种方法可以区分出共同和独特的特征，并有选择地将它们整合到生成的输出中。 我们的方法在部分解耦的对比语言-图像预训练（CLIP）嵌入空间内操作（来自IP适配器），从而能够灵活地转移纹理、形状、运动、风格以及更抽象的概念转化，而无需额外的训练或文本提示。我们在包括样式转换、形态变化和概念转化在内的多种任务上展示了该技术的有效性，并证明了如何将微妙或抽象的属性（例如笔触风格、流线型线条和动态感）无缝地结合到新的图像中。 在一项用户研究中，参与者准确识别出哪些特征是意欲转移的。视觉概念融合方法因其简单性、灵活性以及高层次的控制能力，在艺术、设计和内容创作等创意领域尤为有价值，这些领域需要将多来源灵感中的特定视觉质量进行组合。
URL
https://arxiv.org/abs/2503.21277
PDF
-
StyledStreets: Multi-style Street Simulator with Spatial and Temporal Consistency
2025-03-27 02:52:29Yuyin Chen, Yida Wang, Xueyang Zhang, Kun Zhan, Peng Jia, Yifei Zhan, Xianpeng LangAbstract
Urban scene reconstruction requires modeling both static infrastructure and dynamic elements while supporting diverse environmental conditions. We present \textbf{StyledStreets}, a multi-style street simulator that achieves instruction-driven scene editing with guaranteed spatial and temporal consistency. Building on a state-of-the-art Gaussian Splatting framework for street scenarios enhanced by our proposed pose optimization and multi-view training, our method enables photorealistic style transfers across seasons, weather conditions, and camera setups through three key innovations: First, a hybrid embedding scheme disentangles persistent scene geometry from transient style attributes, allowing realistic environmental edits while preserving structural integrity. Second, uncertainty-aware rendering mitigates supervision noise from diffusion priors, enabling robust training across extreme style variations. Third, a unified parametric model prevents geometric drift through regularized updates, maintaining multi-view consistency across seven vehicle-mounted cameras. Our framework preserves the original scene's motion patterns and geometric relationships. Qualitative results demonstrate plausible transitions between diverse conditions (snow, sandstorm, night), while quantitative evaluations show state-of-the-art geometric accuracy under style transfers. The approach establishes new capabilities for urban simulation, with applications in autonomous vehicle testing and augmented reality systems requiring reliable environmental consistency. Codes will be publicly available upon publication.
Abstract (translated)
城市场景重建需要同时建模静态基础设施和动态元素，并支持各种环境条件。我们介绍了**StyledStreets**，这是一个多风格街景模拟器，它通过保证空间和时间一致性来实现指令驱动的场景编辑。我们的方法基于最先进的高斯点绘框架进行街景建模，并在此基础上提出了姿态优化和多视角训练，使模型能够跨季节、天气条件以及摄像机设置进行逼真的风格转换。以下是三个关键创新： 1. **混合嵌入方案**：该方案将持久的场景几何与瞬时的样式属性分离，允许在保留结构完整性的前提下对环境做出真实感的编辑。 2. **不确定性感知渲染**：这种方法减轻了来自扩散先验的监督噪声，使得模型能够在极端风格变化的情况下进行稳健训练。 3. **统一参数化模型**：通过正则化更新防止几何漂移，并保持七个车载摄像头视角的一致性。 我们的框架保留了原始场景的运动模式和几何关系。定性的结果展示了在不同条件（如雪天、沙尘暴、夜晚）之间合理的过渡，而定量评估显示，在风格转换过程中达到了最先进的几何准确性。该方法为城市模拟建立了一系列新能力，适用于自动驾驶汽车测试以及需要可靠环境一致性的增强现实系统。相关代码将在论文发表后公开提供。
URL
https://arxiv.org/abs/2503.21104
PDF
-
Text-Driven Voice Conversion via Latent State-Space Modeling
2025-03-26 21:30:29Wen Li, Sofia Martinez, Priyanka ShahAbstract
Text-driven voice conversion allows customization of speaker characteristics and prosodic elements using textual descriptions. However, most existing methods rely heavily on direct text-to-speech training, limiting their flexibility in controlling nuanced style elements or timbral features. In this paper, we propose a novel \textbf{Latent State-Space} approach for text-driven voice conversion (\textbf{LSS-VC}). Our method treats each utterance as an evolving dynamical system in a continuous latent space. Drawing inspiration from mamba, which introduced a state-space model for efficient text-driven \emph{image} style transfer, we adapt a loosely related methodology for \emph{voice} style transformation. Specifically, we learn a voice latent manifold where style and content can be manipulated independently by textual style prompts. We propose an adaptive cross-modal fusion mechanism to inject style information into the voice latent representation, enabling interpretable and fine-grained control over speaker identity, speaking rate, and emphasis. Extensive experiments show that our approach significantly outperforms recent baselines in both subjective and objective quality metrics, while offering smoother transitions between styles, reduced artifacts, and more precise text-based style control.
Abstract (translated)
文本驱动的语音转换技术允许通过文本描述自定义说话人的特征和韵律元素。然而，大多数现有的方法严重依赖于直接的文本到语音训练，这限制了它们在控制细微风格要素或音色特性方面的灵活性。本文提出了一种新颖的方法——**隐含状态空间**（Latent State-Space）文本驱动语音转换（LSS-VC）。我们的方法将每个语句视为连续隐含空间中的演化动力系统。借鉴mamba引入的状态空间模型，该模型用于高效的文本驱动图像风格转移，我们将其相关技术适应于声音风格变换中。 具体而言，我们学习了一个可以独立操控的音色潜在流形，其中通过文本风格提示来改变样式和内容。我们提出了一种自适应跨模态融合机制，将风格信息注入到语音潜在表示中，从而实现对说话者身份、语速和强调程度的解释性和精细控制。 大量的实验表明，我们的方法在主观和客观质量指标上均显著优于近期基线方法，并且提供了更平滑的风格转换过程，减少了伪影，并实现了更为精确的文本基础风格控制。
URL
https://arxiv.org/abs/2503.20999
PDF
-
ReverBERT: A State Space Model for Efficient Text-Driven Speech Style Transfer
2025-03-26 21:11:17Michael Brown, Sofia Martinez, Priya SinghAbstract
Text-driven speech style transfer aims to mold the intonation, pace, and timbre of a spoken utterance to match stylistic cues from text descriptions. While existing methods leverage large-scale neural architectures or pre-trained language models, the computational costs often remain high. In this paper, we present \emph{ReverBERT}, an efficient framework for text-driven speech style transfer that draws inspiration from a state space model (SSM) paradigm, loosely motivated by the image-based method of Wang and Liu~\cite{wang2024stylemamba}. Unlike image domain techniques, our method operates in the speech space and integrates a discrete Fourier transform of latent speech features to enable smooth and continuous style modulation. We also propose a novel \emph{Transformer-based SSM} layer for bridging textual style descriptors with acoustic attributes, dramatically reducing inference time while preserving high-quality speech characteristics. Extensive experiments on benchmark speech corpora demonstrate that \emph{ReverBERT} significantly outperforms baselines in terms of naturalness, expressiveness, and computational efficiency. We release our model and code publicly to foster further research in text-driven speech style transfer.
Abstract (translated)
文本驱动的语音风格转换旨在根据文本描述中的风格提示调整口语表达的情感、语速和音色。虽然现有的方法依赖大规模神经网络架构或预训练的语言模型，但计算成本依然较高。在本文中，我们提出了**ReverBERT**，这是一种高效的基于文本的语音风格转换框架，它借鉴了状态空间模型（SSM）范式的思想，受到Wang和Liu提出的基于图像的方法StyleMamba的启发。与图像领域的技术不同，我们的方法直接作用于语音领域，并通过整合潜在语音特征的离散傅里叶变换来实现平滑且连续的风格调节。我们还提出了一种新的**Transformer基SSM层**，用于连接文本风格描述和声学属性，这在保证高质量语音特性的同时显著减少了推理时间。广泛基于基准语音语料库进行的实验表明，ReverBERT在自然度、表现力及计算效率方面明显优于基础模型。我们公开发布了我们的模型和代码，以促进文本驱动的语音风格转换领域的进一步研究。
URL
https://arxiv.org/abs/2503.20992
PDF
-
Pluggable Style Representation Learning for Multi-Style Transfer
2025-03-26 09:44:40Hongda Liu, Longguang Wang, Weijun Guan, Ye Zhang, Yulan GuoAbstract
Due to the high diversity of image styles, the scalability to various styles plays a critical role in real-world applications. To accommodate a large amount of styles, previous multi-style transfer approaches rely on enlarging the model size while arbitrary-style transfer methods utilize heavy backbones. However, the additional computational cost introduced by more model parameters hinders these methods to be deployed on resource-limited devices. To address this challenge, in this paper, we develop a style transfer framework by decoupling the style modeling and transferring. Specifically, for style modeling, we propose a style representation learning scheme to encode the style information into a compact representation. Then, for style transferring, we develop a style-aware multi-style transfer network (SaMST) to adapt to diverse styles using pluggable style representations. In this way, our framework is able to accommodate diverse image styles in the learned style representations without introducing additional overhead during inference, thereby maintaining efficiency. Experiments show that our style representation can extract accurate style information. Moreover, qualitative and quantitative results demonstrate that our method achieves state-of-the-art performance in terms of both accuracy and efficiency. The codes are available in this https URL.
Abstract (translated)
由于图像风格的多样性非常高，因此在实际应用中，能够适应各种风格的可扩展性起着关键作用。为了应对大量的不同风格，以往的多风格转移方法依赖于扩大模型规模，而任意风格转移方法则使用重型骨干网络（backbone）。然而，由更多模型参数引入的额外计算成本阻碍了这些方法在资源受限设备上的部署。 为了解决这一挑战，在本文中我们开发了一个通过解耦样式建模和转换来实现样式的迁移框架。具体来说，在样式建模方面，我们提出了一种风格表示学习方案，用以将风格信息编码成紧凑的表示形式。然后，对于样式转移，我们开发了一种基于可插拔样式表示的感知多风格传输网络（SaMST），以此适应多样化的样式需求。通过这种方式，我们的框架能够在不增加推理过程额外开销的情况下容纳各种图像样式，并保持了效率。 实验表明，我们的样式表示能够准确地提取出风格信息。此外，定性和定量结果均显示，我们的方法在精度和效率方面都达到了最先进的性能水平。代码可以在提供的链接中找到：[此 https URL]。
URL
https://arxiv.org/abs/2503.20368
PDF
-
Color Conditional Generation with Sliced Wasserstein Guidance
2025-03-24 18:06:03Alexander Lobashev, Maria Larchenko, Dmitry GuskovAbstract
We propose SW-Guidance, a training-free approach for image generation conditioned on the color distribution of a reference image. While it is possible to generate an image with fixed colors by first creating an image from a text prompt and then applying a color style transfer method, this approach often results in semantically meaningless colors in the generated image. Our method solves this problem by modifying the sampling process of a diffusion model to incorporate the differentiable Sliced 1-Wasserstein distance between the color distribution of the generated image and the reference palette. Our method outperforms state-of-the-art techniques for color-conditional generation in terms of color similarity to the reference, producing images that not only match the reference colors but also maintain semantic coherence with the original text prompt. Our source code is available at this https URL.
Abstract (translated)
我们提出了一种名为SW-Guidance的方法，这是一种无需训练的图像生成方法，可以根据参考图像的颜色分布进行条件化。尽管可以通过先从文本提示创建图像然后再应用颜色风格迁移方法来生成固定颜色的图像，但这种方法通常会导致在生成的图像中出现语义上无意义的颜色。我们的方法通过修改扩散模型的采样过程来解决这一问题，该过程引入了生成图像与参考调色板之间可微分的切片1-沃瑟斯坦距离（Sliced 1-Wasserstein distance）。我们的方法在颜色相似性方面优于最先进的颜色条件生成技术，并且不仅匹配参考颜色，还能保持与原始文本提示的语义一致性。我们源代码的链接在此：[此URL]。（请将"[此URL]"替换为实际提供的具体网址）
URL
https://arxiv.org/abs/2503.19034
PDF
-
Advancing Cross-Organ Domain Generalization with Test-Time Style Transfer and Diversity Enhancement
2025-03-24 11:22:27Biwen Meng, Xi Long, Wanrong Yang, Ruochen Liu, Yi Tian, Yalin Zheng, Jingxin LiuAbstract
Deep learning has made significant progress in addressing challenges in various fields including computational pathology (CPath). However, due to the complexity of the domain shift problem, the performance of existing models will degrade, especially when it comes to multi-domain or cross-domain tasks. In this paper, we propose a Test-time style transfer (T3s) that uses a bidirectional mapping mechanism to project the features of the source and target domains into a unified feature space, enhancing the generalization ability of the model. To further increase the style expression space, we introduce a Cross-domain style diversification module (CSDM) to ensure the orthogonality between style bases. In addition, data augmentation and low-rank adaptation techniques are used to improve feature alignment and sensitivity, enabling the model to adapt to multi-domain inputs effectively. Our method has demonstrated effectiveness on three unseen datasets.
Abstract (translated)
深度学习在包括计算病理学（CPath）在内的多个领域取得了显著进展。然而，由于领域迁移问题的复杂性，现有模型的表现会下降，特别是在处理多域或跨域任务时尤为明显。在这篇论文中，我们提出了一种测试时间风格转换（T3s），该方法采用双向映射机制将源域和目标域的特征投影到统一的特征空间中，从而增强模型的泛化能力。为了进一步扩大风格表达空间，我们引入了跨域风格多样化模块（CSDM）来确保风格基线之间的正交性。此外，我们还使用数据增强和低秩自适应技术来改进特征对齐和敏感度，使模型能够有效应对多域输入。我们的方法在三个未见过的数据集上展示了其有效性。
URL
https://arxiv.org/abs/2503.18567
PDF
-
SaMam: Style-aware State Space Model for Arbitrary Image Style Transfer
2025-03-20 08:18:27Hongda Liu, Longguang Wang, Ye Zhang, Ziru Yu, Yulan GuoAbstract
Global effective receptive field plays a crucial role for image style transfer (ST) to obtain high-quality stylized results. However, existing ST backbones (e.g., CNNs and Transformers) suffer huge computational complexity to achieve global receptive fields. Recently, the State Space Model (SSM), especially the improved variant Mamba, has shown great potential for long-range dependency modeling with linear complexity, which offers a approach to resolve the above dilemma. In this paper, we develop a Mamba-based style transfer framework, termed SaMam. Specifically, a mamba encoder is designed to efficiently extract content and style information. In addition, a style-aware mamba decoder is developed to flexibly adapt to various styles. Moreover, to address the problems of local pixel forgetting, channel redundancy and spatial discontinuity of existing SSMs, we introduce both local enhancement and zigzag scan. Qualitative and quantitative results demonstrate that our SaMam outperforms state-of-the-art methods in terms of both accuracy and efficiency.
Abstract (translated)
全局有效的感受野在图像风格转换（ST）中扮演着关键角色，以获得高质量的风格化结果。然而，现有的风格转移骨干网络（如CNN和Transformer），为了实现全局感受野，面临着巨大的计算复杂度问题。最近，状态空间模型（SSM），特别是改进版的Mamba，在长距离依赖建模方面表现出线性复杂度的巨大潜力，为解决上述难题提供了可能的方法。本文中，我们开发了一个基于Mamba的风格转换框架，称为SaMam。具体而言，设计了一种高效的mamba编码器来提取内容和样式信息，并且开发了一种感知样式的mamba解码器以灵活适应各种样式。此外，为了应对现有SSMs中存在的局部像素遗忘、通道冗余以及空间不连续性问题，我们引入了局部增强技术和之字形扫描技术。定性和定量的结果都表明，我们的SaMam框架在准确性和效率方面均优于当前最先进的方法。
URL
https://arxiv.org/abs/2503.15934
PDF
-
Uncertainty-Aware Diffusion Guided Refinement of 3D Scenes
2025-03-19 23:14:27Sarosij Bose, Arindam Dutta, Sayak Nag, Junge Zhang, Jiachen Li, Konstantinos Karydis, Amit K. Roy ChowdhuryAbstract
Reconstructing 3D scenes from a single image is a fundamentally ill-posed task due to the severely under-constrained nature of the problem. Consequently, when the scene is rendered from novel camera views, existing single image to 3D reconstruction methods render incoherent and blurry views. This problem is exacerbated when the unseen regions are far away from the input camera. In this work, we address these inherent limitations in existing single image-to-3D scene feedforward networks. To alleviate the poor performance due to insufficient information beyond the input image's view, we leverage a strong generative prior in the form of a pre-trained latent video diffusion model, for iterative refinement of a coarse scene represented by optimizable Gaussian parameters. To ensure that the style and texture of the generated images align with that of the input image, we incorporate on-the-fly Fourier-style transfer between the generated images and the input image. Additionally, we design a semantic uncertainty quantification module that calculates the per-pixel entropy and yields uncertainty maps used to guide the refinement process from the most confident pixels while discarding the remaining highly uncertain ones. We conduct extensive experiments on real-world scene datasets, including in-domain RealEstate-10K and out-of-domain KITTI-v2, showing that our approach can provide more realistic and high-fidelity novel view synthesis results compared to existing state-of-the-art methods.
Abstract (translated)
从单张图像重建三维场景是一个本质上未定义的问题，因为该问题的约束条件极为不足。因此，当使用现有的一图转三维重构方法从新颖视角渲染场景时，生成的画面通常是不连贯且模糊不清的，尤其是对于远离输入图像视点的未见区域而言，这一问题更为严重。在这项工作中，我们针对现有单张图像到三维场景前馈网络中存在的固有限制进行了改进。 为了缓解由于缺乏超出输入图像视野的信息而导致的表现不佳的问题，我们利用了一种强大的生成先验知识的形式——预训练的潜在视频扩散模型，以迭代的方式优化粗略场景表示中的可调优高斯参数。为了确保生成图像的风格和纹理与输入图像一致，我们在生成的图像与输入图像之间进行了实时傅里叶样式传输。 此外，我们设计了一个语义不确定性量化模块，该模块计算每个像素的熵值，并据此产生用于指导从最自信像素开始优化过程的同时排除高度不确定像素的不确定性地图。 通过在包括领域内的RealEstate-10K和跨领域的KITTI-v2在内的真实场景数据集上进行广泛实验，我们展示了我们的方法能够提供比现有最先进的技术更逼真且高质量的新视角合成结果。
URL
https://arxiv.org/abs/2503.15742
PDF
