Style_Transfer
-
GuitarFlow: Realistic Electric Guitar Synthesis From Tablatures via Flow Matching and Style Transfer
2025-10-23 13:31:41Jackson Loth, Pedro Sarmento, Mark Sandler, Mathieu BarthetAbstract
Music generation in the audio domain using artificial intelligence (AI) has witnessed steady progress in recent years. However for some instruments, particularly the guitar, controllable instrument synthesis remains limited in expressivity. We introduce GuitarFlow, a model designed specifically for electric guitar synthesis. The generative process is guided using tablatures, an ubiquitous and intuitive guitar-specific symbolic format. The tablature format easily represents guitar-specific playing techniques (e.g. bends, muted strings and legatos), which are more difficult to represent in other common music notation formats such as MIDI. Our model relies on an intermediary step of first rendering the tablature to audio using a simple sample-based virtual instrument, then performing style transfer using Flow Matching in order to transform the virtual instrument audio into more realistic sounding examples. This results in a model that is quick to train and to perform inference, requiring less than 6 hours of training data. We present the results of objective evaluation metrics, together with a listening test, in which we show significant improvement in the realism of the generated guitar audio from tablatures.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21872
PDF
-
Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
2025-10-20 19:46:02Jiajun Fan, Tong Wei, Chaoran Cheng, Yuxin Chen, Ge LiuAbstract
Balancing exploration and exploitation during reinforcement learning fine-tuning of generative models presents a critical challenge, as existing approaches rely on fixed divergence regularization that creates an inherent dilemma: strong regularization preserves model capabilities but limits reward optimization, while weak regularization enables greater alignment but risks instability or reward hacking. We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), which automatically adjusts regularization strength based on advantage estimates-reducing regularization for high-value samples while applying stronger regularization to poor samples, enabling policies to navigate between exploration and aggressive exploitation according to data quality. Our implementation with Wasserstein-2 regularization for flow matching generative models achieves remarkable results on text-to-image generation, achieving better semantic alignment and diversity than offline methods like DPO and online methods with fixed regularization like ORW-CFM-W2. ADRPO enables a 2B parameter SD3 model to surpass much larger models with 4.8B and 12B parameters in attribute binding, semantic consistency, artistic style transfer, and compositional control while maintaining generation diversity. ADRPO generalizes to KL-regularized fine-tuning of both text-only LLMs and multi-modal reasoning models, enhancing existing online RL methods like GRPO. In LLM fine-tuning, ADRPO demonstrates an emergent ability to escape local optima through active exploration, while in multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning, enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, offering an effective plug-and-play solution to the exploration-exploitation challenge across diverse generative architectures and modalities.
Abstract (translated)
在强化学习（Reinforcement Learning，RL）中对生成模型进行微调时，平衡探索和利用是一个关键挑战。现有的方法依赖于固定的发散正则化（divergence regularization），这种固定的方法产生了固有的两难局面：强烈的正则化可以保持模型的能力但限制了奖励优化的范围；而较弱的正则化虽然能更好地对齐目标，却可能带来不稳定或奖励操纵的风险。为此，我们提出了自适应发散正则化的策略优化（Adaptive Divergence Regularized Policy Optimization, ADRPO），该方法可以根据优势估计自动调整正则化强度——对于高价值样本减少正则化，而对于低质量的样本施加强正则化，从而使得策略能够在探索与积极利用之间根据数据的质量灵活切换。 我们的实现方案中使用了Wasserstein-2正则化的流匹配生成模型，在文本到图像生成的任务上取得了显著成果，其在语义对齐和多样性方面超越了离线方法（如DPO）以及固定正则化在线方法（例如ORW-CFM-W2）。ADRPO允许一个参数量为20亿的SD3模型在属性绑定、语义一致性、艺术风格转换及组合控制等任务上优于具有48亿和120亿参数的更大规模模型，同时保持生成多样性的平衡。 此外，ADRPO适用于KL正则化微调仅限于文本的大语言模型（Large Language Models, LLMs）以及多模态推理模型。它增强了现有在线RL方法（如GRPO）。在LLM微调过程中，ADRPO展示了通过积极探索逃离局部最优解的能力；而在处理多模态音频推理时，相比GRPO，它能凭借更出色的逐步推理能力击败包括Gemini 2.5 Pro和GPT-4o Audio在内的大型商业模型。这使得一个具有70亿参数的模型能够超越这些商业模型，并为各种生成架构及模态中探索与利用之间的挑战提供了一个有效的即插即用解决方案。 综上所述，ADRPO通过自适应调整正则化强度的方式，在多个场景下提升了生成模型的效果和性能，特别是在平衡探索与利用方面表现出色。
URL
https://arxiv.org/abs/2510.18053
PDF
-
REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting
2025-10-18 08:53:08Changyue Shi, Minghao Chen, Yiping Mao, Chuxiao Yang, Xinyuan Hu, Jiajun Ding, Zhou YuAbstract
Bridging the gap between complex human instructions and precise 3D object grounding remains a significant challenge in vision and robotics. Existing 3D segmentation methods often struggle to interpret ambiguous, reasoning-based instructions, while 2D vision-language models that excel at such reasoning lack intrinsic 3D spatial understanding. In this paper, we introduce REALM, an innovative MLLM-agent framework that enables open-world reasoning-based segmentation without requiring extensive 3D-specific post-training. We perform segmentation directly on 3D Gaussian Splatting representations, capitalizing on their ability to render photorealistic novel views that are highly suitable for MLLM comprehension. As directly feeding one or more rendered views to the MLLM can lead to high sensitivity to viewpoint selection, we propose a novel Global-to-Local Spatial Grounding strategy. Specifically, multiple global views are first fed into the MLLM agent in parallel for coarse-level localization, aggregating responses to robustly identify the target object. Then, several close-up novel views of the object are synthesized to perform fine-grained local segmentation, yielding accurate and consistent 3D masks. Extensive experiments show that REALM achieves remarkable performance in interpreting both explicit and implicit instructions across LERF, 3D-OVS, and our newly introduced REALM3D benchmarks. Furthermore, our agent framework seamlessly supports a range of 3D interaction tasks, including object removal, replacement, and style transfer, demonstrating its practical utility and versatility. Project page: this https URL.
Abstract (translated)
在视觉和机器人技术领域，将复杂的指令与精确的三维物体定位相结合仍然是一个重大挑战。现有的三维分割方法通常难以解释那些基于推理的模糊指令，而二维视觉语言模型虽然擅长处理此类推理任务，却缺乏对三维空间的理解。为此，在这篇论文中，我们提出了REALM框架，这是一种创新的多模态大规模预训练语言模型（MLLM）与代理相结合的方法，它能够在无需进行大量特定于3D数据的后期训练的情况下执行基于推理的开放世界分割。我们在3D高斯光束表示上直接进行分割，利用其能够渲染非常逼真的新视角的特点，这些视角非常适合MLLM的理解能力。 然而，将一个或多个渲染视图直接输入到MLLM中可能会导致对视点选择的高度敏感性。因此，我们提出了一个新颖的全局到局部空间定位策略：首先并行地向代理中的MLLM提供多个全局视图以进行粗略级别的本地化，并通过聚合这些响应来稳健地识别目标物体；然后，生成该对象的一系列特写的新视角来进行精细分割，从而获得精确且一致的3D掩膜。 经过广泛的实验表明，REALM在理解LERF、3D-OVS以及我们新引入的REALM3D基准测试中的明确和隐含指令方面表现出卓越性能。此外，我们的代理框架还能够无缝支持一系列三维交互任务，包括物体移除、替换及风格转移等操作，这展示了其实用性和灵活性。 项目主页: [链接](this https URL)
URL
https://arxiv.org/abs/2510.16410
PDF
-
Generative human motion mimicking through feature extraction in denoising diffusion settings
2025-10-16 14:41:54Alexander Okupnik, Johannes Schneider, Kyriakos FlourisAbstract
Recent success with large language models has sparked a new wave of verbal human-AI interaction. While such models support users in a variety of creative tasks, they lack the embodied nature of human interaction. Dance, as a primal form of human expression, is predestined to complement this experience. To explore creative human-AI interaction exemplified by dance, we build an interactive model based on motion capture (MoCap) data. It generates an artificial other by partially mimicking and also "creatively" enhancing an incoming sequence of movement data. It is the first model, which leverages single-person motion data and high level features in order to do so and, thus, it does not rely on low level human-human interaction data. It combines ideas of two diffusion models, motion inpainting, and motion style transfer to generate movement representations that are both temporally coherent and responsive to a chosen movement reference. The success of the model is demonstrated by quantitatively assessing the convergence of the feature distribution of the generated samples and the test set which serves as simulating the human performer. We show that our generations are first steps to creative dancing with AI as they are both diverse showing various deviations from the human partner while appearing realistic.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00011
PDF
-
LLM one-shot style transfer for Authorship Attribution and Verification
2025-10-15 08:43:24Pablo Miralles-Gonz\'alez, Javier Huertas-Tato, Alejandro Mart\'in, David CamachoAbstract
Computational stylometry analyzes writing style through quantitative patterns in text, supporting applications from forensic tasks such as identity linking and plagiarism detection to literary attribution in the humanities. Supervised and contrastive approaches rely on data with spurious correlations and often confuse style with topic. Despite their natural use in AI-generated text detection, the CLM pre-training of modern LLMs has been scarcely leveraged for general authorship problems. We propose a novel unsupervised approach based on this extensive pre-training and the in-context learning capabilities of LLMs, employing the log-probabilities of an LLM to measure style transferability from one text to another. Our method significantly outperforms LLM prompting approaches of comparable scale and achieves higher accuracy than contrastively trained baselines when controlling for topical correlations. Moreover, performance scales fairly consistently with the size of the base model and, in the case of authorship verification, with an additional mechanism that increases test-time computation; enabling flexible trade-offs between computational cost and accuracy.
Abstract (translated)
计算性文体测量通过文本中的定量模式分析写作风格，支持从法医任务（如身份链接和抄袭检测）到人文学科中文学归属等各类应用。虽然监督学习和对比方法依赖于带有伪相关的数据，并且常常将写作风格与主题混淆，但现代大型语言模型（LLM）的CLM预训练在一般作者识别问题上却鲜少被利用。我们提出了一种基于广泛预训练及LLM上下文学习能力的新颖无监督方法，通过使用LLM的日志概率来测量一种文本向另一种文本的写作风格可转移性。我们的方法显著优于规模相当的提示式LLM方法，并且在控制主题相关性的前提下，比对比训练基线获得了更高的准确率。此外，性能随着基础模型大小的增长而相对一致地提升；而在作者验证的情况下，通过增加测试时计算量的额外机制，使计算成本与准确性之间的灵活权衡成为可能。
URL
https://arxiv.org/abs/2510.13302
PDF
-
Uncertainty-Aware ControlNet: Bridging Domain Gaps with Synthetic Image Generation
2025-10-13 12:41:28Joshua Niemeijer, Jan Ehrhardt, Heinz Handels, Hristina UzunovaAbstract
Generative Models are a valuable tool for the controlled creation of high-quality image data. Controlled diffusion models like the ControlNet have allowed the creation of labeled distributions. Such synthetic datasets can augment the original training distribution when discriminative models, like semantic segmentation, are trained. However, this augmentation effect is limited since ControlNets tend to reproduce the original training distribution. This work introduces a method to utilize data from unlabeled domains to train ControlNets by introducing the concept of uncertainty into the control mechanism. The uncertainty indicates that a given image was not part of the training distribution of a downstream task, e.g., segmentation. Thus, two types of control are engaged in the final network: an uncertainty control from an unlabeled dataset and a semantic control from the labeled dataset. The resulting ControlNet allows us to create annotated data with high uncertainty from the target domain, i.e., synthetic data from the unlabeled distribution with labels. In our scenario, we consider retinal OCTs, where typically high-quality Spectralis images are available with given ground truth segmentations, enabling the training of segmentation networks. The recent development in Home-OCT devices, however, yields retinal OCTs with lower quality and a large domain shift, such that out-of-the-pocket segmentation networks cannot be applied for this type of data. Synthesizing annotated images from the Home-OCT domain using the proposed approach closes this gap and leads to significantly improved segmentation results without adding any further supervision. The advantage of uncertainty-guidance becomes obvious when compared to style transfer: it enables arbitrary domain shifts without any strict learning of an image style. This is also demonstrated in a traffic scene experiment.
Abstract (translated)
生成模型是用于控制高质量图像数据创建的宝贵工具。受控扩散模型，如ControlNet，允许创建标记分布。这样的合成数据集可以在训练辨别性模型（例如语义分割）时扩充原始训练分布。然而，这种增强效果有限，因为ControlNets倾向于复制原训练数据分布。这项工作引入了一种方法，利用未标注领域的数据来训练ControlNets，并通过将不确定性概念引入控制机制中实现这一目标。不确定性表示给定的图像并非下游任务（如分割）原始训练数据集的一部分。因此，在最终网络中有两种类型的控制：一种是从未标记数据集中来的不确定性控制和一种是从标记数据集来的语义控制。由此产生的ControlNet使我们能够从目标领域创建具有高不确定性的注释数据，即带有标签的来自未标注分布的合成图像。 在我们的场景中考虑视网膜OCT（光学相干断层扫描），通常有高质量的Spectralis图像及其给定的真实分割结果，这可以用来训练分割网络。然而，最近家用OCT设备的发展带来了较低质量和域偏移较大的视网膜OCT数据，使得现成的分割网络无法应用于此类数据。利用所提出的方法从Home-OCT领域生成注释图像填补了这一空白，并在不添加额外监督的情况下显著提高了分割结果。 与风格转移相比，不确定性指导的优势变得明显：它可以实现任意域偏移而不严格学习图像样式。这一点也在交通场景实验中得到了验证。
URL
https://arxiv.org/abs/2510.11346
PDF
-
SceneTextStylizer: A Training-Free Scene Text Style Transfer Framework with Diffusion Model
2025-10-13 02:11:57Honghui Yuan, Keiji YanaiAbstract
With the rapid development of diffusion models, style transfer has made remarkable progress. However, flexible and localized style editing for scene text remains an unsolved challenge. Although existing scene text editing methods have achieved text region editing, they are typically limited to content replacement and simple styles, which lack the ability of free-style transfer. In this paper, we introduce SceneTextStylizer, a novel training-free diffusion-based framework for flexible and high-fidelity style transfer of text in scene images. Unlike prior approaches that either perform global style transfer or focus solely on textual content modification, our method enables prompt-guided style transformation specifically for text regions, while preserving both text readability and stylistic consistency. To achieve this, we design a feature injection module that leverages diffusion model inversion and self-attention to transfer style features effectively. Additionally, a region control mechanism is introduced by applying a distance-based changing mask at each denoising step, enabling precise spatial control. To further enhance visual quality, we incorporate a style enhancement module based on the Fourier transform to reinforce stylistic richness. Extensive experiments demonstrate that our method achieves superior performance in scene text style transformation, outperforming existing state-of-the-art methods in both visual fidelity and text preservation.
Abstract (translated)
随着扩散模型的快速发展，风格迁移取得了显著进展。然而，场景文本中的灵活且局部化的风格编辑仍然是一个未解决的挑战。尽管现有的场景文本编辑方法已经实现了文字区域编辑，但它们通常局限于内容替换和简单的样式更改，并缺乏自由式风格转移的能力。在本文中，我们介绍了SceneTextStylizer，这是一种新颖的、无需训练的基于扩散模型的框架，用于对场景图像中的文本进行灵活且高保真的风格转换。 与先前的方法要么执行全局风格迁移，要么仅专注于文本内容修改不同，我们的方法通过使用提示引导的方式实现了针对文字区域的特定风格变换，并同时保持了文本可读性和风格一致性。为了实现这一点，我们设计了一个特征注入模块，该模块利用扩散模型逆向工程和自注意力机制有效地转移风格特征。此外，我们引入了一种基于距离变化掩码的区域控制机制，在每次去噪步骤中应用以实现精确的空间控制。为了进一步提高视觉质量，我们还集成了一个基于傅立叶变换的风格增强模块，用于加强风格丰富性。 大量的实验表明，我们的方法在场景文本风格转换方面表现出了优越性能，在视觉保真度和文本保留方面均优于现有的最先进的方法。
URL
https://arxiv.org/abs/2510.10910
PDF
-
Jigsaw3D: Disentangled 3D Style Transfer via Patch Shuffling and Masking
2025-10-12 08:22:57Yuteng Ye, Zheng Zhang, Qinchuan Zhang, Di Wang, Youjia Zhang, Wenxiao Zhang, Wei Yang, Yuan LiuAbstract
Controllable 3D style transfer seeks to restyle a 3D asset so that its textures match a reference image while preserving the integrity and multi-view consistency. The prevalent methods either rely on direct reference style token injection or score-distillation from 2D diffusion models, which incurs heavy per-scene optimization and often entangles style with semantic content. We introduce Jigsaw3D, a multi-view diffusion based pipeline that decouples style from content and enables fast, view-consistent stylization. Our key idea is to leverage the jigsaw operation - spatial shuffling and random masking of reference patches - to suppress object semantics and isolate stylistic statistics (color palettes, strokes, textures). We integrate these style cues into a multi-view diffusion model via reference-to-view cross-attention, producing view-consistent stylized renderings conditioned on the input mesh. The renders are then style-baked onto the surface to yield seamless textures. Across standard 3D stylization benchmarks, Jigsaw3D achieves high style fidelity and multi-view consistency with substantially lower latency, and generalizes to masked partial reference stylization, multi-object scene styling, and tileable texture generation. Project page is available at: this https URL
Abstract (translated)
可控的3D风格转换旨在重新设计一个3D资产，使其纹理与参考图像相匹配，并同时保持其完整性和多视角一致性。目前流行的方法要么依赖于直接引用样式令牌注入或从2D扩散模型进行评分蒸馏，这需要大量的场景级优化，且经常将风格与语义内容纠缠在一起。 我们引入了Jigsaw3D，这是一种基于多视图扩散的流水线，它可以将风格和内容分离，并实现快速、视角一致性的风格化。我们的核心思想是利用拼图操作（参考补丁的空间打乱和随机掩码）来抑制物体语义并隔离样式统计信息（调色板、笔触、纹理）。我们通过视图交叉注意力机制将这些样式提示集成到多视图扩散模型中，生成以输入网格为条件的视角一致化的风格化渲染。然后将这些渲染中的样式烘焙到表面，以产生无缝的纹理。 在标准的3D风格化基准测试中，Jigsaw3D实现了高风格保真度和多视角一致性，并且延迟明显降低。此外，该方法还能推广至部分遮挡参考样式的风格化、多个物体场景的风格化以及可平铺纹理生成等应用场景。项目页面可在以下链接访问：[此URL](this%20https%20URL)（请将占位符替换为实际链接）。
URL
https://arxiv.org/abs/2510.10497
PDF
-
ReMix: Towards a Unified View of Consistent Character Generation and Editing
2025-10-11 10:31:56Benjia Zhou, Bin Fu, Pei Cheng, Yanru Wang, Jiayuan Fan, Tao ChenAbstract
Recent advances in large-scale text-to-image diffusion models (e.g., FLUX.1) have greatly improved visual fidelity in consistent character generation and editing. However, existing methods rarely unify these tasks within a single framework. Generation-based approaches struggle with fine-grained identity consistency across instances, while editing-based methods often lose spatial controllability and instruction alignment. To bridge this gap, we propose ReMix, a unified framework for character-consistent generation and editing. It constitutes two core components: the ReMix Module and IP-ControlNet. The ReMix Module leverages the multimodal reasoning ability of MLLMs to edit semantic features of input images and adapt instruction embeddings to the native DiT backbone without fine-tuning. While this ensures coherent semantic layouts, pixel-level consistency and pose controllability remain challenging. To address this, IP-ControlNet extends ControlNet to decouple semantic and layout cues from reference images and introduces an {\epsilon}-equivariant latent space that jointly denoises the reference and target images within a shared noise space. Inspired by convergent evolution and quantum decoherence,i.e., where environmental noise drives state convergence, this design promotes feature alignment in the hidden space, enabling consistent object generation while preserving identity. ReMix supports a wide range of tasks, including personalized generation, image editing, style transfer, and multi-condition synthesis. Extensive experiments validate its effectiveness and efficiency as a unified framework for character-consistent image generation and editing.
Abstract (translated)
最近在大规模文本到图像扩散模型（例如FLUX.1）方面的进展，极大地提升了连贯人物生成和编辑的视觉保真度。然而，现有的方法很少将这些任务整合在一个单一框架内完成。基于生成的方法难以实现实例间的细微身份一致性，而基于编辑的方法则往往失去空间控制性和指令对齐性。为解决这一问题，我们提出了ReMix框架，旨在统一处理人物连贯性的生成和编辑任务。该框架包含两个核心组件：ReMix模块和IP-ControlNet。 - **ReMix模块**利用多模态推理能力来编辑输入图像的语义特征，并将指令嵌入到DiT骨干网络中，而无需进行微调。尽管这确保了连贯的语义布局，但像素级一致性以及姿态控制仍然具有挑战性。 - **IP-ControlNet**则在ControlNet的基础上进行了扩展，以从参考图像中解耦出语义和布局线索，并引入了一个{\epsilon}-等变隐空间，该空间能在共享噪声空间内同时对齐参考图和目标图的去噪过程。这种设计灵感来源于趋同进化与量子退相干现象，即环境中的噪声驱动状态收敛。这一设计在隐藏空间中促进了特征的对齐，在保持身份一致性的前提下实现了物体生成的一致性。 ReMix框架支持多种任务，包括个性化生成、图像编辑、风格迁移和多条件合成等，并通过广泛的实验验证了其作为统一框架的有效性和效率，在连贯人物图像生成和编辑领域表现卓越。
URL
https://arxiv.org/abs/2510.10156
PDF
-
UniVideo: Unified Understanding, Generation, and Editing for Videos
2025-10-09 16:01:30Cong Wei, Quande Liu, Zixuan Ye, Qiulin Wang, Xintao Wang, Pengfei Wan, Kun Gai, Wenhu ChenAbstract
Unified multimodal models have shown promising results in multimodal content generation and editing but remain largely limited to the image domain. In this work, we present UniVideo, a versatile framework that extends unified modeling to the video domain. UniVideo adopts a dual-stream design, combining a Multimodal Large Language Model (MLLM) for instruction understanding with a Multimodal DiT (MMDiT) for video generation. This design enables accurate interpretation of complex multimodal instructions while preserving visual consistency. Built on this architecture, UniVideo unifies diverse video generation and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVideo matches or surpasses state-of-the-art task-specific baselines in text/image-to-video generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combining editing with style transfer, by integrating multiple capabilities within a single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data to this setting, handling unseen instructions such as green-screening characters or changing materials within a video. Beyond these core capabilities, UniVideo also supports visual-prompt-based video generation, where the MLLM interprets visual prompts and guides the MMDiT during synthesis. To foster future research, we will release our model and code.
Abstract (translated)
统一的多模态模型在多模态内容生成和编辑方面显示出了令人鼓舞的结果，但仍然主要局限于图像领域。在这项工作中，我们介绍了UniVideo，这是一个灵活的框架，它将统一建模扩展到了视频领域。UniVideo采用了双流设计，结合了用于指令理解的多模态大型语言模型（MLLM）与用于视频生成的多模态DiT（MMDiT）。这种设计能够准确解释复杂的多模态指令，并保持视觉一致性。基于这一架构，UniVideo将多样化的视频生成和编辑任务统一在一个单一的多模态指令范式下，并在这些任务之间进行联合训练。大量的实验表明，UniVideo在文本/图像到视频生成、上下文视频生成以及上下文视频编辑方面达到了或超越了最先进的特定任务基线。 值得注意的是，UniVideo的统一设计能够实现两种形式的泛化：首先，UniVideo支持任务组合，如将编辑与风格迁移相结合，在单一指令中集成多种能力。其次，即使没有在自由形式视频编辑上进行明确训练，UniVideo也能将其编辑能力从大规模图像编辑数据迁移到这种环境中，处理诸如绿幕人物或改变视频中材质等未见过的指令。 除了这些核心功能外，UniVideo还支持基于视觉提示的视频生成，在这种模式下，MLLM解释视觉提示并指导MMDiT在合成过程中。为了促进未来的研究，我们将发布我们的模型和代码。
URL
https://arxiv.org/abs/2510.08377
PDF
-
PickStyle: Video-to-Video Style Transfer with Context-Style Adapters
2025-10-08 21:02:55Soroush Mehraban, Vida Adeli, Jacob Rommann, Babak Taati, Kyryl TruskovskyiAbstract
We address the task of video style transfer with diffusion models, where the goal is to preserve the context of an input video while rendering it in a target style specified by a text prompt. A major challenge is the lack of paired video data for supervision. We propose PickStyle, a video-to-video style transfer framework that augments pretrained video diffusion backbones with style adapters and benefits from paired still image data with source-style correspondences for training. PickStyle inserts low-rank adapters into the self-attention layers of conditioning modules, enabling efficient specialization for motion-style transfer while maintaining strong alignment between video content and style. To bridge the gap between static image supervision and dynamic video, we construct synthetic training clips from paired images by applying shared augmentations that simulate camera motion, ensuring temporal priors are preserved. In addition, we introduce Context-Style Classifier-Free Guidance (CS-CFG), a novel factorization of classifier-free guidance into independent text (style) and video (context) directions. CS-CFG ensures that context is preserved in generated video while the style is effectively transferred. Experiments across benchmarks show that our approach achieves temporally coherent, style-faithful, and content-preserving video translations, outperforming existing baselines both qualitatively and quantitatively.
Abstract (translated)
我们研究了使用扩散模型进行视频风格迁移的任务，目标是在保持输入视频上下文的同时将其渲染为由文本提示指定的目标风格。一个主要的挑战是缺乏配对视频数据以供监督训练。为此，我们提出了PickStyle框架，这是一种视频到视频的风格迁移框架，它在预先训练好的视频扩散主干网络中加入了风格适配器，并利用带有源风格对应关系的成对静止图像数据进行训练。PickStyle通过在条件模块的自我注意层中插入低秩适配器来实现高效的运动-风格传输专业化，同时保持视频内容和风格之间的一致性。 为了弥合静态图像监督与动态视频之间的差距，我们通过施加模拟摄像机移动的共享增强处理，从成对静止图象构建合成训练片段，以确保时间先验得以保留。此外，我们引入了上下文-样式无分类器引导（Context-Style Classifier-Free Guidance, CS-CFG），这是一种新颖的将无分类器引导分解为独立的文字（风格）和视频（上下文）方向的方法。CS-CFG确保在生成的视频中保持内容一致性的同时有效迁移风格。 跨基准实验表明，我们的方法实现了时间上连贯、风格忠实且内容保留的视频转换，在定性和定量评估中均优于现有基线方法。
URL
https://arxiv.org/abs/2510.07546
PDF
-
Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement
2025-10-02 07:03:50Jianing Yang, Sheng Li, Takahiro Shinozaki, Yuki Saito, Hiroshi SaruwatariAbstract
Current emotional Text-To-Speech (TTS) and style transfer methods rely on reference encoders to control global style or emotion vectors, but do not capture nuanced acoustic details of the reference speech. To this end, we propose a novel emotional TTS method that enables fine-grained phoneme-level emotion embedding prediction while disentangling intrinsic attributes of the reference speech. The proposed method employs a style disentanglement method to guide two feature extractors, reducing mutual information between timbre and emotion features, and effectively separating distinct style components from the reference speech. Experimental results demonstrate that our method outperforms baseline TTS systems in generating natural and emotionally rich speech. This work highlights the potential of disentangled and fine-grained representations in advancing the quality and flexibility of emotional TTS systems.
Abstract (translated)
当前的情感文本到语音（TTS）和风格转换方法依赖于参考编码器来控制全局风格或情感向量，但无法捕捉参考语音中的细微声学细节。为此，我们提出了一种新的情感TTS方法，该方法能够在分离参考语音的内在属性的同时，在音素级别上进行精细的情感嵌入预测。所提出的方法采用了一种风格解耦方法，指导两个特征提取器减少音色和情感特征之间的相互信息，并有效地从参考语音中区分出不同的风格成分。实验结果表明，与基线TTS系统相比，我们的方法在生成自然且情感丰富的语音方面表现更佳。这项工作强调了在提高情感TTS系统的质量和灵活性方面，解耦的细粒度表示法的巨大潜力。
URL
https://arxiv.org/abs/2510.01722
PDF
-
PyramidStyler: Transformer-Based Neural Style Transfer with Pyramidal Positional Encoding and Reinforcement Learning
2025-10-02 06:54:52Raahul Krishna Durairaju (California State University, Fullerton), K. Saruladha (Puducherry Technological University)Abstract
Neural Style Transfer (NST) has evolved from Gatys et al.'s (2015) CNN-based algorithm, enabling AI-driven artistic image synthesis. However, existing CNN and transformer-based models struggle to scale efficiently to complex styles and high-resolution inputs. We introduce PyramidStyler, a transformer framework with Pyramidal Positional Encoding (PPE): a hierarchical, multi-scale encoding that captures both local details and global context while reducing computational load. We further incorporate reinforcement learning to dynamically optimize stylization, accelerating convergence. Trained on Microsoft COCO and WikiArt, PyramidStyler reduces content loss by 62.6% (to 2.07) and style loss by 57.4% (to 0.86) after 4000 epochs--achieving 1.39 s inference--and yields further improvements (content 2.03; style 0.75) with minimal speed penalty (1.40 s) when using RL. These results demonstrate real-time, high-quality artistic rendering, with broad applications in media and design.
Abstract (translated)
神经风格迁移（NST）源自 Gatys 等人于2015年提出的基于 CNN 的算法，该技术使得 AI 驱动的艺术图像合成成为可能。然而，现有的基于 CNN 和变压器的模型在处理复杂风格和高分辨率输入时难以高效扩展。我们引入了 PyramidStyler，这是一种采用金字塔位置编码（PPE）的变压器框架：这种分层、多尺度编码方法能够在减少计算负担的同时捕捉局部细节与全局背景信息。此外，我们还结合了强化学习来动态优化图像美化效果，从而加快收敛速度。PyramidStyler 在训练过程中使用了微软 COCO 和 WikiArt 数据集，在经过4000个周期的训练后，其内容损失降低了62.6%（降至2.07），风格损失则减少了57.4%（降至0.86），并在实现1.39秒推理时间的同时达到了上述效果。当采用强化学习时，PyramidStyler 还能在几乎不增加运行速度的情况下进一步优化结果（内容损耗为2.03；风格损耗为0.75，且仅需额外的0.01秒）。这些成果表明了实时高质量艺术渲染的可能性，并在媒体和设计领域有着广泛的应用前景。
URL
https://arxiv.org/abs/2510.01715
PDF
-
Multi-level Dynamic Style Transfer for NeRFs
2025-10-01 07:19:27Zesheng Li, Shuaibo Li, Wei Ma, Jianwei Guo, Hongbin ZhaAbstract
As the application of neural radiance fields (NeRFs) in various 3D vision tasks continues to expand, numerous NeRF-based style transfer techniques have been developed. However, existing methods typically integrate style statistics into the original NeRF pipeline, often leading to suboptimal results in both content preservation and artistic stylization. In this paper, we present multi-level dynamic style transfer for NeRFs (MDS-NeRF), a novel approach that reengineers the NeRF pipeline specifically for stylization and incorporates an innovative dynamic style injection module. Particularly, we propose a multi-level feature adaptor that helps generate a multi-level feature grid representation from the content radiance field, effectively capturing the multi-scale spatial structure of the scene. In addition, we present a dynamic style injection module that learns to extract relevant style features and adaptively integrates them into the content patterns. The stylized multi-level features are then transformed into the final stylized view through our proposed multi-level cascade decoder. Furthermore, we extend our 3D style transfer method to support omni-view style transfer using 3D style references. Extensive experiments demonstrate that MDS-NeRF achieves outstanding performance for 3D style transfer, preserving multi-scale spatial structures while effectively transferring stylistic characteristics.
Abstract (translated)
随着神经辐射场（NeRFs）在各种三维视觉任务中的应用不断扩展，已经开发出了多种基于NeRF的风格转换技术。然而，现有方法通常将风格统计信息整合到原始的NeRF流程中，这往往导致内容保留和艺术风格化方面的结果不尽如人意。本文介绍了一种名为多级动态风格传输（MDS-NeRF）的新方法，该方法专门重新设计了用于风格化的NeRF管道，并引入了一个创新性的动态样式注入模块。 具体而言，我们提出了一个多级特征适配器，它有助于从内容辐射场中生成一个多级特征网格表示，有效捕捉场景的多尺度空间结构。此外，我们还介绍了一个动态样式注入模块，该模块能够学习提取相关风格特征，并将它们自适应地集成到内容模式中。通过我们的提出的多级级联解码器，这些经过风格化的多级特性随后被转换为最终的风格化视图。 另外，我们将3D风格传输方法扩展以支持使用3D样式参考进行全方位风格传输。大量的实验表明，MDS-NeRF在保持多尺度空间结构的同时能够有效地转移艺术特征，在三维风格传输方面取得了卓越的表现。
URL
https://arxiv.org/abs/2510.00592
PDF
-
Stylos: Multi-View 3D Stylization with Single-Forward Gaussian Splatting
2025-09-30 16:09:13Hanzhou Liu, Jia Huang, Mi Lu, Srikanth Saripalli, Peng JiangAbstract
We present Stylos, a single-forward 3D Gaussian framework for 3D style transfer that operates on unposed content, from a single image to a multi-view collection, conditioned on a separate reference style image. Stylos synthesizes a stylized 3D Gaussian scene without per-scene optimization or precomputed poses, achieving geometry-aware, view-consistent stylization that generalizes to unseen categories, scenes, and styles. At its core, Stylos adopts a Transformer backbone with two pathways: geometry predictions retain self-attention to preserve geometric fidelity, while style is injected via global cross-attention to enforce visual consistency across views. With the addition of a voxel-based 3D style loss that aligns aggregated scene features to style statistics, Stylos enforces view-consistent stylization while preserving geometry. Experiments across multiple datasets demonstrate that Stylos delivers high-quality zero-shot stylization, highlighting the effectiveness of global style-content coupling, the proposed 3D style loss, and the scalability of our framework from single view to large-scale multi-view settings.
Abstract (translated)
我们介绍了Stylos，这是一个单一前向的3D高斯框架，用于在未布置的内容上进行三维风格转换，从单张图像到多视角集合，并以单独的参考风格图像为条件。Stylos 无需场景优化或预计算姿态即可合成具有几何感知和视图一致性的风格化3D高斯场景，从而能够泛化至未知类别、场景和风格。在核心方面，Stylos 采用了一个Transformer骨干网，并包含两个路径：几何预测保留自注意力以保持几何精度，而风格则通过全局跨注意力注入以确保视图间的一致性。随着基于体素的3D风格损失的加入，该损失将聚合场景特征与风格统计对齐，Stylos 在保持几何结构的同时实现了视图一致性的风格化。多数据集上的实验表明，Stylos 能够提供高质量的零样本风格化效果，这突显了全局风格-内容耦合的有效性、提出的3D风格损失以及我们的框架从单视角扩展到大规模多视角设置的能力。
URL
https://arxiv.org/abs/2509.26455
PDF
-
Personalized Scientific Figure Caption Generation: An Empirical Study on Author-Specific Writing Style Transfer
2025-09-30 05:52:15Jaeyoung Kim, Jongho Lee, Hongjun Choi, Sion JangAbstract
We study personalized figure caption generation using author profile data from scientific papers. Our experiments demonstrate that rich author profile data, combined with relevant metadata, can significantly improve the personalization performance of multimodal large language models. However, we also reveal a fundamental trade-off between matching author style and maintaining caption quality. Our findings offer valuable insights and future directions for developing practical caption automation systems that balance both objectives. This work was conducted as part of the 3rd SciCap challenge.
Abstract (translated)
我们研究了利用科学论文中的作者个人资料数据进行个性化图像描述生成的方法。我们的实验表明，丰富的作者个人资料数据与相关元数据相结合，可以显著提高多模态大型语言模型的个性化性能。然而，我们也揭示了在匹配作者风格和保持描述质量之间存在一个基本的权衡。我们的研究发现为开发既能平衡这两个目标又能实际应用的图像描述自动化系统提供了宝贵的见解和未来发展方向。这项工作是作为第三届SciCap挑战赛的一部分完成的。
URL
https://arxiv.org/abs/2509.25817
PDF
-
OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing
2025-09-29 15:11:09Zhihong Chen, Xuehai Bai, Yang Shi, Chaoyou Fu, Huanyu Zhang, Haotian Wang, Xiaoyan Sun, Zhang Zhang, Liang Wang, Yuanxing Zhang, Pengfei Wan, Yi-Fan ZhangAbstract
The performance of unified multimodal models for image generation and editing is fundamentally constrained by the quality and comprehensiveness of their training data. While existing datasets have covered basic tasks like style transfer and simple object manipulation, they often lack the systematic structure and challenging scenarios required for real-world applications. To address this bottleneck, we introduce OpenGPT-4o-Image, a large-scale dataset constructed using a novel methodology that combines hierarchical task taxonomy with automated data generation. Our taxonomy not only includes fundamental capabilities such as text rendering and style control but also introduces highly practical yet challenging categories like scientific imagery for chemistry illustrations and complex instruction editing requiring simultaneous execution of multiple operations. Through an automated pipeline leveraging structured resource pools and GPT-4o, we generate 80k high-quality instruction-image pairs with controlled diversity, covering 11 major domains and 51 subtasks. Extensive experiments show that fine-tuning leading models on our dataset achieves significant performance gains across multiple benchmarks, with improvements of up to 18\% on editing tasks (UniWorld-V1 on ImgEdit-Bench) and 13% on generation tasks (Harmon on GenEval). Our work demonstrates that systematic data construction is key to advancing multimodal AI capabilities.
Abstract (translated)
统一多模态模型在图像生成和编辑方面的性能从根本上受到其训练数据质量和全面性的限制。尽管现有的数据集已经涵盖了风格转移和简单对象操作等基本任务，但它们通常缺乏系统结构和现实应用中所需的挑战性场景。为了解决这一瓶颈问题，我们引入了OpenGPT-4o-Image，这是一个采用新颖方法构建的大规模数据集，该方法结合了层次化的任务分类学与自动数据生成技术。 我们的分类法不仅涵盖了诸如文本渲染和风格控制等基础能力，还引入了一些实用且具有挑战性的类别，例如用于化学插图的科学图像以及需要同时执行多项操作的复杂指令编辑。通过利用结构化资源池和GPT-4o构建的自动化管道，我们生成了80,000对高质量的指令-图像配对，并在控制多样性的情况下涵盖了11个主要领域和51项子任务。 广泛的实验表明，在我们的数据集上进行微调后，领先模型在多个基准测试中取得了显著性能提升：编辑任务中的UniWorld-V1在ImgEdit-Bench上的改进高达18%，而生成任务中Harmon在GenEval上的改进则为13%。我们的研究工作证明了系统化地构建数据对于推进多模态人工智能能力至关重要。
URL
https://arxiv.org/abs/2509.24900
PDF
-
Discovering 'Words' in Music: Unsupervised Learning of Compositional Sparse Code for Symbolic Music
2025-09-29 11:10:57Tianle Wang, Sirui Zhang, Xinyi Tong, Peiyang Yu, Jishang Chen, Liangke Zhao, Xinpu Gao, Yves Zhu, Tiezheng Ge, Bo Zheng, Duo Xu, Yang Liu, Xin Jin, Feng Yu, Songchun ZhuAbstract
This paper presents an unsupervised machine learning algorithm that identifies recurring patterns -- referred to as ``music-words'' -- from symbolic music data. These patterns are fundamental to musical structure and reflect the cognitive processes involved in composition. However, extracting these patterns remains challenging because of the inherent semantic ambiguity in musical interpretation. We formulate the task of music-word discovery as a statistical optimization problem and propose a two-stage Expectation-Maximization (EM)-based learning framework: 1. Developing a music-word dictionary; 2. Reconstructing the music data. When evaluated against human expert annotations, the algorithm achieved an Intersection over Union (IoU) score of 0.61. Our findings indicate that minimizing code length effectively addresses semantic ambiguity, suggesting that human optimization of encoding systems shapes musical semantics. This approach enables computers to extract ``basic building blocks'' from music data, facilitating structural analysis and sparse encoding. The method has two primary applications. First, in AI music, it supports downstream tasks such as music generation, classification, style transfer, and improvisation. Second, in musicology, it provides a tool for analyzing compositional patterns and offers insights into the principle of minimal encoding across diverse musical styles and composers.
Abstract (translated)
本文介绍了一种无监督机器学习算法，该算法能够从符号音乐数据中识别出重复出现的模式——称为“音乐词汇”（music-words）。这些模式是音乐结构的基本组成部分，并反映了作曲过程中的认知过程。然而，由于音乐解释中存在的语义模糊性，提取这些模式仍然是一个挑战。我们把发现音乐词汇的任务形式化为一个统计优化问题，并提出了一种基于两阶段期望最大化（EM）的学习框架：1. 创建音乐词典；2. 重构音乐数据。 在与人类专家标注进行对比评估时，该算法达到了0.61的交并比（IoU）得分。我们的研究结果表明，通过最小化编码长度可以有效地解决语义模糊性问题，暗示了人类优化编码系统对音乐语义的影响。这种方法使计算机能够从音乐数据中提取“基本构建模块”，从而促进结构分析和稀疏编码。 该方法有两个主要应用领域：首先，在AI音乐方面，它支持下游任务如音乐生成、分类、风格转换及即兴创作；其次，在音乐学研究中，它提供了一种用于分析作曲模式的工具，并为不同音乐流派及作曲家中的最小化编码原则提供了见解。
URL
https://arxiv.org/abs/2509.24603
PDF
-
Diffusion Bridge or Flow Matching? A Unifying Framework and Comparative Analysis
2025-09-29 09:45:22Kaizhen Zhu, Mokai Pan, Zhechuan Yu, Jingya Wang, Jingyi Yu, Ye ShiAbstract
Diffusion Bridge and Flow Matching have both demonstrated compelling empirical performance in transformation between arbitrary distributions. However, there remains confusion about which approach is generally preferable, and the substantial discrepancies in their modeling assumptions and practical implementations have hindered a unified theoretical account of their relative merits. We have, for the first time, provided a unified theoretical and experimental validation of these two models. We recast their frameworks through the lens of Stochastic Optimal Control and prove that the cost function of the Diffusion Bridge is lower, guiding the system toward more stable and natural trajectories. Simultaneously, from the perspective of Optimal Transport, interpolation coefficients $t$ and $1-t$ of Flow Matching become increasingly ineffective when the training data size is reduced. To corroborate these theoretical claims, we propose a novel, powerful architecture for Diffusion Bridge built on a latent Transformer, and implement a Flow Matching model with the same structure to enable a fair performance comparison in various experiments. Comprehensive experiments are conducted across Image Inpainting, Super-Resolution, Deblurring, Denoising, Translation, and Style Transfer tasks, systematically varying both the distributional discrepancy (different difficulty) and the training data size. Extensive empirical results align perfectly with our theoretical predictions and allow us to delineate the respective advantages and disadvantages of these two models. Our code is available at this https URL.
Abstract (translated)
扩散桥（Diffusion Bridge）和流匹配（Flow Matching）在任意分布之间的转换中都展示了令人信服的实证性能。然而，关于哪一种方法更优的选择仍然存在疑惑，并且这两种模型之间巨大的建模假设差异及实际实现上的不同也阻碍了对它们相对优势进行统一理论阐述的可能性。我们首次提供了这两种模型统合理论和实验验证的方法。我们将这两个框架重新构想为随机最优控制的视角，并证明扩散桥的成本函数较低，这有助于系统趋向更稳定且自然的轨迹变化。同时，从最优传输的角度来看，在训练数据量减少的情况下，流匹配中的插值系数 $t$ 和 $1-t$ 逐渐变得无效。 为了验证这些理论主张，我们提出了一种新颖而强大的基于潜在Transformer架构的扩散桥模型，并实现了一个结构相同的流匹配模型以在各种实验中进行公平性能比较。我们在图像修复、超分辨率、去模糊化、去噪、转换以及风格迁移等任务上进行了全面实验，系统地改变分布差异（不同难度）和训练数据量。广泛的实证结果与我们的理论预测完全一致，并使我们能够明确指出这两种模型各自的优缺点。 有关该工作的代码可以在提供的链接中获取。
URL
https://arxiv.org/abs/2509.24531
PDF
-
Disentangling Score Content and Performance Style for Joint Piano Rendering and Transcription
2025-09-28 13:36:33Wei Zeng, Junchuan Zhao, Ye WangAbstract
Expressive performance rendering (EPR) and automatic piano transcription (APT) are fundamental yet inverse tasks in music information retrieval: EPR generates expressive performances from symbolic scores, while APT recovers scores from performances. Despite their dual nature, prior work has addressed them independently. In this paper we propose a unified framework that jointly models EPR and APT by disentangling note-level score content and global performance style representations from both paired and unpaired data. Our framework is built on a transformer-based sequence-to-sequence architecture and is trained using only sequence-aligned data, without requiring fine-grained note-level alignment. To automate the rendering process while ensuring stylistic compatibility with the score, we introduce an independent diffusion-based performance style recommendation module that generates style embeddings directly from score content. This modular component supports both style transfer and flexible rendering across a range of expressive styles. Experimental results from both objective and subjective evaluations demonstrate that our framework achieves competitive performance on EPR and APT tasks, while enabling effective content-style disentanglement, reliable style transfer, and stylistically appropriate rendering. Demos are available at this https URL
Abstract (translated)
以下是该段落的中文翻译： 表现性表演渲染（EPR）和自动钢琴转录（APT）是音乐信息检索中的基本且互为逆向的任务：EPR 从符号乐谱生成具有表现力的演奏，而 APT 则是从演奏中恢复乐谱。尽管它们的本质相反，之前的研究工作分别独立地处理这些问题。在本文中，我们提出了一种统一框架，通过分离配对和非配对数据中的音符级乐谱内容和全局表演风格表示来共同建模 EPR 和 APT。我们的框架基于转换器（transformer）的序列到序列架构，并仅使用序列对齐的数据进行训练，无需精细的音符级别对齐。为了在确保与乐谱风格一致的情况下自动化渲染过程，我们引入了一个独立的基于扩散模型的表演风格推荐模块，该模块直接从乐谱内容生成风格嵌入。这一模块化组件支持跨多种表现性风格的内容转换和灵活渲染。实验结果表明，在客观评估和主观评估中，我们的框架在 EPR 和 APT 任务上实现了具有竞争力的表现，并且能够有效地实现内容与风格的分离、可靠的风格转移以及符合风格的渲染。演示可在该链接查看：[此链接处应填写实际提供的演示网址]
URL
https://arxiv.org/abs/2509.23878
PDF
