Style_Transfer
-
B'ezier Meets Diffusion: Robust Generation Across Domains for Medical Image Segmentation
2025-09-26 15:23:17Chen Li, Meilong Xu, Xiaoling Hu, Weimin Lyu, Chao ChenAbstract
Training robust learning algorithms across different medical imaging modalities is challenging due to the large domain gap. Unsupervised domain adaptation (UDA) mitigates this problem by using annotated images from the source domain and unlabeled images from the target domain to train the deep models. Existing approaches often rely on GAN-based style transfer, but these methods struggle to capture cross-domain mappings in regions with high variability. In this paper, we propose a unified framework, Bézier Meets Diffusion, for cross-domain image generation. First, we introduce a Bézier-curve-based style transfer strategy that effectively reduces the domain gap between source and target domains. The transferred source images enable the training of a more robust segmentation model across domains. Thereafter, using pseudo-labels generated by this segmentation model on the target domain, we train a conditional diffusion model (CDM) to synthesize high-quality, labeled target-domain images. To mitigate the impact of noisy pseudo-labels, we further develop an uncertainty-guided score matching method that improves the robustness of CDM training. Extensive experiments on public datasets demonstrate that our approach generates realistic labeled images, significantly augmenting the target domain and improving segmentation performance.
Abstract (translated)
在不同医学影像模态上训练鲁棒的学习算法具有挑战性，因为存在较大的领域差距。无监督域适应（UDA）通过使用源领域的标注图像和目标领域的未标注图像来训练深度模型，从而缓解了这一问题。现有方法通常依赖于基于GAN的样式迁移技术，但这些方法难以捕捉高变异区域中的跨域映射关系。 在本文中，我们提出了一种统一框架，“贝塞尔遇见扩散”（Bézier Meets Diffusion），用于跨领域图像生成。首先，我们引入了一个基于贝塞尔曲线的样式迁移策略，该策略有效地减少了源领域和目标领域之间的差距。迁移后的源域图像使我们能够在多个领域中训练出更稳健的分割模型。随后，利用此分割模型在目标领域的伪标签，我们训练一个条件扩散模型（CDM）来合成高质量、带标签的目标领域图像。 为了减轻噪声伪标签的影响，我们进一步开发了一种基于不确定性的得分匹配方法以提高CDM训练过程中的鲁棒性。在公共数据集上的大量实验表明，我们的方法能够生成逼真的标注图像，显著扩充了目标域，并提高了分割性能。
URL
https://arxiv.org/abs/2509.22476
PDF
-
GemDetox at TextDetox CLEF 2025: Enhancing a Massively Multilingual Model for Text Detoxification on Low-resource Languages
2025-09-24 10:06:40Trung Duc Anh Dang, Ferdinando Pio D'EliaAbstract
As social-media platforms emerge and evolve faster than the regulations meant to oversee them, automated detoxification might serve as a timely tool for moderators to enforce safe discourse at scale. We here describe our submission to the PAN 2025 Multilingual Text Detoxification Challenge, which rewrites toxic single-sentence inputs into neutral paraphrases across 15 typologically diverse languages. Building on a 12B-parameter Gemma-3 multilingual transformer, we apply parameter-efficient LoRA SFT fine-tuning and prompting techniques like few-shot and Chain-of-Thought. Our multilingual training corpus combines 3,600 human-authored parallel pairs, 21,600 machine-translated synthetic pairs, and model-generated pairs filtered by Jaccard thresholds. At inference, inputs are enriched with three LaBSE-retrieved neighbors and explicit toxic-span annotations. Evaluated via Style Transfer Accuracy, LaBSE-based semantic preservation, and xCOMET fluency, our system ranks first on high-resource and low-resource languages. Ablations show +0.081 joint score increase from few-shot examples and +0.088 from basic CoT prompting. ANOVA analysis identifies language resource status as the strongest predictor of performance ($\eta^2$ = 0.667, p < 0.01).
Abstract (translated)
随着社交媒体平台的快速涌现和演变，监管措施往往跟不上步伐。在这种情况下，自动化去毒工具可以作为管理员在大规模范围内实施安全对话的重要手段。在这里，我们介绍了我们在PAN 2025多语言文本去毒挑战中的参赛作品，该作品将有毒的单句输入重写为15种类型多样化的语言中性短语。 我们的解决方案基于一个参数量达12B的Gemma-3多语言转换器，并应用了高效的LoRA SFT微调和如少样本学习（few-shot learning）及链式思维（Chain-of-Thought）等提示技术。我们使用的多语言训练数据集结合了3,600对由人类编写的平行文本、21,600对机器翻译的合成文本，以及通过杰卡德阈值过滤后生成的模型文本对。 在推理阶段，输入会使用LaBSE检索的三个邻居进行扩展，并加上明确的有毒片段注释。我们的系统在风格转换准确率（Style Transfer Accuracy）、基于LaBSE的语义保存能力和xCOMET流畅度方面进行了评估，在高资源和低资源语言中均排名第一。 消融研究表明，少样本示例可使联合得分增加0.081，而基本链式思维提示则使分数增加了0.088。方差分析（ANOVA）发现，语言资源状况是预测性能最强的因素（$\eta^2$ = 0.667, p < 0.01）。
URL
https://arxiv.org/abs/2510.01250
PDF
-
One-shot Embroidery Customization via Contrastive LoRA Modulation
2025-09-23 12:58:15Jun Ma, Qian He, Gaofeng He, Huang Chen, Chen Liu, Xiaogang Jin, Huamin WangAbstract
Diffusion models have significantly advanced image manipulation techniques, and their ability to generate photorealistic images is beginning to transform retail workflows, particularly in presale visualization. Beyond artistic style transfer, the capability to perform fine-grained visual feature transfer is becoming increasingly important. Embroidery is a textile art form characterized by intricate interplay of diverse stitch patterns and material properties, which poses unique challenges for existing style transfer methods. To explore the customization for such fine-grained features, we propose a novel contrastive learning framework that disentangles fine-grained style and content features with a single reference image, building on the classic concept of image analogy. We first construct an image pair to define the target style, and then adopt a similarity metric based on the decoupled representations of pretrained diffusion models for style-content separation. Subsequently, we propose a two-stage contrastive LoRA modulation technique to capture fine-grained style features. In the first stage, we iteratively update the whole LoRA and the selected style blocks to initially separate style from content. In the second stage, we design a contrastive learning strategy to further decouple style and content through self-knowledge distillation. Finally, we build an inference pipeline to handle image or text inputs with only the style blocks. To evaluate our method on fine-grained style transfer, we build a benchmark for embroidery customization. Our approach surpasses prior methods on this task and further demonstrates strong generalization to three additional domains: artistic style transfer, sketch colorization, and appearance transfer.
Abstract (translated)
扩散模型在图像处理技术方面取得了显著进展，它们生成逼真照片的能力已经开始改变零售工作流程，特别是在预售可视化领域。除了艺术风格转换外，执行细粒度视觉特征转移的能力变得越来越重要。刺绣是一种纺织艺术品形式，以其多样的针法图案和材料特性复杂交织为特点，这对现有的风格迁移方法提出了独特挑战。为了探索对这种细粒度特性的定制，我们提出了一种新颖的对比学习框架，在单个参考图像的基础上分离出细粒度的风格与内容特征，基于经典的概念——图像类比。首先，我们构建一对图象来定义目标风格，然后采用基于预训练扩散模型解耦表示的相似性指标来进行风格和内容的分离。随后，我们提出了一种两阶段对比LoRA调制技术，以捕捉细粒度的风格特征。在第一阶段，我们迭代更新整个LoRA及选定的风格模块来初步分离风格与内容。在第二阶段，我们设计了一个基于自知识蒸馏的对比学习策略，进一步解耦风格和内容。最后，我们建立了一条推理管线，仅使用风格块就可以处理图像或文本输入。 为了评估我们的方法在细粒度风格转换上的性能，我们为刺绣定制建立了基准测试。我们的方法在此任务上超越了先前的方法，并进一步展示了其在三个额外领域中的强大泛化能力：艺术风格转移、草图着色和外观变换。
URL
https://arxiv.org/abs/2509.18948
PDF
-
DA-Font: Few-Shot Font Generation via Dual-Attention Hybrid Integration
2025-09-20 11:12:15Weiran Chen, Guiqian Zhu, Ying Li, Yi Ji, Chunping LiuAbstract
Few-shot font generation aims to create new fonts with a limited number of glyph references. It can be used to significantly reduce the labor cost of manual font design. However, due to the variety and complexity of font styles, the results generated by existing methods often suffer from visible defects, such as stroke errors, artifacts and blurriness. To address these issues, we propose DA-Font, a novel framework which integrates a Dual-Attention Hybrid Module (DAHM). Specifically, we introduce two synergistic attention blocks: the component attention block that leverages component information from content images to guide the style transfer process, and the relation attention block that further refines spatial relationships through interacting the content feature with both original and stylized component-wise representations. These two blocks collaborate to preserve accurate character shapes and stylistic textures. Moreover, we also design a corner consistency loss and an elastic mesh feature loss to better improve geometric alignment. Extensive experiments show that our DA-Font outperforms the state-of-the-art methods across diverse font styles and characters, demonstrating its effectiveness in enhancing structural integrity and local fidelity. The source code can be found at \href{this https URL}{\textit{this https URL}}.
Abstract (translated)
少样本字体生成旨在使用有限数量的字符参考来创建新字体。这种方法可以显著降低手动设计字体的工作量。然而，由于字体风格的变化和复杂性，现有方法生成的结果往往存在明显的缺陷，如笔画错误、伪影和模糊。为了解决这些问题，我们提出了DA-Font框架，该框架集成了双注意力混合模块（DAHM）。具体来说，我们引入了两个协同的注意力块：组件注意块通过利用内容图像中的部件信息来指导风格转换过程；关系注意块进一步通过交互原始和样式化的部件级表示来细化空间关系。这两个模块协作以保持准确的文字形状和风格纹理。 此外，我们还设计了一致性角损失和弹性网格特征损失，以便更好地改善几何对齐。广泛的实验表明，在各种字体风格和字符中，我们的DA-Font方法优于最先进的方法，证明了其在增强结构完整性和局部保真度方面的有效性。源代码可以在 [此链接](this https URL) 获取。
URL
https://arxiv.org/abs/2509.16632
PDF
-
MultiEdit: Advancing Instruction-based Image Editing on Diverse and Challenging Tasks
2025-09-18 05:33:38Mingsong Li, Lin Liu, Hongjun Wang, Haoxing Chen, Xijun Gu, Shizhan Liu, Dong Gong, Junbo Zhao, Zhenzhong Lan, Jianguo LiAbstract
Current instruction-based image editing (IBIE) methods struggle with challenging editing tasks, as both editing types and sample counts of existing datasets are limited. Moreover, traditional dataset construction often contains noisy image-caption pairs, which may introduce biases and limit model capabilities in complex editing scenarios. To address these limitations, we introduce MultiEdit, a comprehensive dataset featuring over 107K high-quality image editing samples. It encompasses 6 challenging editing tasks through a diverse collection of 18 non-style-transfer editing types and 38 style transfer operations, covering a spectrum from sophisticated style transfer to complex semantic operations like person reference editing and in-image text editing. We employ a novel dataset construction pipeline that utilizes two multi-modal large language models (MLLMs) to generate visual-adaptive editing instructions and produce high-fidelity edited images, respectively. Extensive experiments demonstrate that fine-tuning foundational open-source models with our MultiEdit-Train set substantially improves models' performance on sophisticated editing tasks in our proposed MultiEdit-Test benchmark, while effectively preserving their capabilities on the standard editing benchmark. We believe MultiEdit provides a valuable resource for advancing research into more diverse and challenging IBIE capabilities. Our dataset is available at this https URL.
Abstract (translated)
当前基于指令的图像编辑（IBIE）方法在处理复杂的编辑任务时面临挑战，因为现有数据集中的编辑类型和样本数量都有限。此外，传统的数据集构建方式通常包含有噪声的图片-描述对，这可能会引入偏差并限制模型在复杂编辑场景中的能力。为了解决这些局限性，我们推出了MultiEdit，这是一个全面的数据集，包含了超过107,000个高质量的图像编辑样本。该数据集涵盖了6项具有挑战性的编辑任务，包括通过多样化的18种非风格迁移编辑类型和38种风格转移操作收集而成的一系列复杂语义操作，例如人物参考编辑和图片内文本编辑等。 我们采用了一种新颖的数据构建管道，利用两种多模态大型语言模型（MLLMs）来生成适应视觉的编辑指令，并产生高保真的编辑图像。通过广泛的实验表明，在我们的MultiEdit-Test基准测试中对基础开源模型使用MultiEdit-Train数据集进行微调能够显著提高模型在复杂编辑任务上的表现，同时有效保持它们在标准编辑基准测试中的能力。 我们认为，MultiEdit为推进研究以实现更多样化和具有挑战性的IBIE功能提供了宝贵资源。我们的数据集可以在以下链接获取：[此处插入实际的URL]。
URL
https://arxiv.org/abs/2509.14638
PDF
-
ColonCrafter: A Depth Estimation Model for Colonoscopy Videos Using Diffusion Priors
2025-09-16 20:40:22Romain Hardy, Tyler Berzin, Pranav RajpurkarAbstract
Three-dimensional (3D) scene understanding in colonoscopy presents significant challenges that necessitate automated methods for accurate depth estimation. However, existing depth estimation models for endoscopy struggle with temporal consistency across video sequences, limiting their applicability for 3D reconstruction. We present ColonCrafter, a diffusion-based depth estimation model that generates temporally consistent depth maps from monocular colonoscopy videos. Our approach learns robust geometric priors from synthetic colonoscopy sequences to generate temporally consistent depth maps. We also introduce a style transfer technique that preserves geometric structure while adapting real clinical videos to match our synthetic training domain. ColonCrafter achieves state-of-the-art zero-shot performance on the C3VD dataset, outperforming both general-purpose and endoscopy-specific approaches. Although full trajectory 3D reconstruction remains a challenge, we demonstrate clinically relevant applications of ColonCrafter, including 3D point cloud generation and surface coverage assessment.
Abstract (translated)
在结肠镜检查中的三维（3D）场景理解面临着重大挑战，需要自动化的方法来准确估计深度。然而，现有的用于内窥镜的深度估计模型在视频序列的时间一致性方面存在问题，限制了它们在3D重建中的应用效果。我们提出了ColonCrafter，这是一种基于扩散过程的深度估计模型，可以从单目结肠镜检查视频中生成时间一致性的深度图。我们的方法从合成的结肠镜序列中学习到鲁棒的几何先验，以生成时间一致性好的深度地图。此外，我们还引入了一种风格迁移技术，可以在保留几何结构的同时将实际临床视频适配到我们的合成训练领域。 ColonCrafter在C3VD数据集上实现了最先进的零样本性能，优于通用模型和专门针对内窥镜的方法。尽管完整的轨迹3D重建仍然是一个挑战，但我们展示了ColonCrafter的一些具有临床意义的应用，包括生成3D点云和评估表面覆盖情况。
URL
https://arxiv.org/abs/2509.13525
PDF
-
Domain Adaptive SAR Wake Detection: Leveraging Similarity Filtering and Memory Guidance
2025-09-14 08:35:39He Gao, Baoxiang Huang, Milena Radenkovic, Borui Li, Ge ChenAbstract
Synthetic Aperture Radar (SAR), with its all- weather and wide-area observation capabilities, serves as a crucial tool for wake detection. However, due to its complex imaging mechanism, wake features in SAR images often appear abstract and noisy, posing challenges for accurate annotation. In contrast, optical images provide more distinct visual cues, but models trained on optical data suffer from performance degradation when applied to SAR images due to domain shift. To address this cross-modal domain adaptation challenge, we propose a Similarity-Guided and Memory-Guided Domain Adap- tation (termed SimMemDA) framework for unsupervised domain adaptive ship wake detection via instance-level feature similarity filtering and feature memory guidance. Specifically, to alleviate the visual discrepancy between optical and SAR images, we first utilize WakeGAN to perform style transfer on optical images, generating pseudo-images close to the SAR style. Then, instance-level feature similarity filtering mechanism is designed to identify and prioritize source samples with target-like dis- tributions, minimizing negative transfer. Meanwhile, a Feature- Confidence Memory Bank combined with a K-nearest neighbor confidence-weighted fusion strategy is introduced to dynamically calibrate pseudo-labels in the target domain, improving the reliability and stability of pseudo-labels. Finally, the framework further enhances generalization through region-mixed training, strategically combining source annotations with calibrated tar- get pseudo-labels. Experimental results demonstrate that the proposed SimMemDA method can improve the accuracy and robustness of cross-modal ship wake detection tasks, validating the effectiveness and feasibility of the proposed method.
Abstract (translated)
合成孔径雷达（SAR）凭借其全天候和广域观测能力，成为舰船尾流检测的重要工具。然而，由于其复杂的成像机制，SAR图像中的尾流特征往往显得抽象且嘈杂，这给准确标注带来了挑战。相比之下，光学影像提供了更为清晰的视觉线索，但基于光学数据训练出的模型在应用于SAR影像时会因领域差异而性能下降。为了应对跨模态领域的适应性挑战，我们提出了一种以相似度引导和记忆引导的域自适应框架（简称SimMemDA），用于通过实例级特征相似度过滤及特征记忆引导来实现无监督的跨域舰船尾流检测。具体来说，为了解决光学与SAR影像之间的视觉差异问题，我们首先利用WakeGAN对光学图像进行风格转换，生成接近于SAR样式的真实伪图。随后设计了实例级别的特征相似性过滤机制以识别并优先选择分布类似于目标领域的源样本，从而减少负向迁移现象的发生。同时引入结合K近邻权重融合策略的特征置信度记忆库，用于动态校准目标领域中的伪标签，提高其可靠性和稳定性。最后，该框架通过区域混合训练进一步提升了模型的泛化能力，将源标注与校准后的目标伪标签有策略地结合起来。实验结果表明所提出的SimMemDA方法能够提升跨模态舰船尾流检测任务的准确度和鲁棒性，验证了其有效性和可行性。
URL
https://arxiv.org/abs/2509.12279
PDF
-
PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptation
2025-09-14 05:05:27Zeyu Dong, Yuyang Yin, Yuqi Li, Eric Li, Hao-Xiang Guo, Yikai WangAbstract
Generating high-quality 360° panoramic videos remains a significant challenge due to the fundamental differences between panoramic and traditional perspective-view projections. While perspective videos rely on a single viewpoint with a limited field of view, panoramic content requires rendering the full surrounding environment, making it difficult for standard video generation models to adapt. Existing solutions often introduce complex architectures or large-scale training, leading to inefficiency and suboptimal results. Motivated by the success of Low-Rank Adaptation (LoRA) in style transfer tasks, we propose treating panoramic video generation as an adaptation problem from perspective views. Through theoretical analysis, we demonstrate that LoRA can effectively model the transformation between these projections when its rank exceeds the degrees of freedom in the task. Our approach efficiently fine-tunes a pretrained video diffusion model using only approximately 1,000 videos while achieving high-quality panoramic generation. Experimental results demonstrate that our method maintains proper projection geometry and surpasses previous state-of-the-art approaches in visual quality, left-right consistency, and motion diversity.
Abstract (translated)
生成高质量的360°全景视频仍然是一个重大挑战，因为全景和传统透视视图投影之间存在根本差异。传统的视角视频依赖于单一视点且视野有限，而全景内容则需要渲染整个周围环境，这使得标准视频生成模型难以适应这种需求。现有的解决方案通常会引入复杂的架构或大规模训练，导致效率低下并产生次优结果。受低秩适配（LoRA）在风格转移任务中成功的启发，我们提出将全景视频生成视为从透视视角进行转换的问题。通过理论分析，我们证明当LoRA的秩超过任务中的自由度时，它可以有效地建模这两种投影之间的转换。我们的方法可以高效地微调预训练的视频扩散模型，并仅使用大约1,000个视频就能实现高质量全景视频生成。实验结果表明，该方法在保持正确的投影几何的同时，在视觉质量、左右一致性以及运动多样性方面超越了现有的最先进方法。
URL
https://arxiv.org/abs/2509.11092
PDF
-
LADB: Latent Aligned Diffusion Bridges for Semi-Supervised Domain Translation
2025-09-10 14:23:07Xuqin Wang, Tao Wu, Yanfeng Zhang, Lu Liu, Dong Wang, Mingwei Sun, Yongliang Wang, Niclas Zeller, Daniel CremersAbstract
Diffusion models excel at generating high-quality outputs but face challenges in data-scarce domains, where exhaustive retraining or costly paired data are often required. To address these limitations, we propose Latent Aligned Diffusion Bridges (LADB), a semi-supervised framework for sample-to-sample translation that effectively bridges domain gaps using partially paired data. By aligning source and target distributions within a shared latent space, LADB seamlessly integrates pretrained source-domain diffusion models with a target-domain Latent Aligned Diffusion Model (LADM), trained on partially paired latent representations. This approach enables deterministic domain mapping without the need for full supervision. Compared to unpaired methods, which often lack controllability, and fully paired approaches that require large, domain-specific datasets, LADB strikes a balance between fidelity and diversity by leveraging a mixture of paired and unpaired latent-target couplings. Our experimental results demonstrate superior performance in depth-to-image translation under partial supervision. Furthermore, we extend LADB to handle multi-source translation (from depth maps and segmentation masks) and multi-target translation in a class-conditioned style transfer task, showcasing its versatility in handling diverse and heterogeneous use cases. Ultimately, we present LADB as a scalable and versatile solution for real-world domain translation, particularly in scenarios where data annotation is costly or incomplete.
Abstract (translated)
扩散模型在生成高质量输出方面表现出色，但在数据稀疏领域却面临挑战，这些领域往往需要进行详尽的再训练或耗费大量资金获取配对数据。为解决这些问题，我们提出了一个半监督框架——潜空间对齐扩散桥接（Latent Aligned Diffusion Bridges, LADB），该框架通过使用部分配对的数据有效地弥合不同域之间的差距，用于样本到样本的转换任务。LADB 通过在共享的潜在空间中对齐源域和目标域分布，无缝地将预训练的源域扩散模型与基于部分配对潜在表示的目标域潜空间对齐扩散模型（Latent Aligned Diffusion Model, LADM）结合在一起。这种方法能够在无需全面监督的情况下实现确定性的领域映射。 与缺乏可控性的非配对方法相比，以及需要大量特定领域数据集的完全配对方法相比，LADB 通过利用配对和未配对潜在目标耦合的混合方式，在保真度和多样性之间取得了平衡。我们的实验结果表明，在部分监督下的深度到图像转换任务中，LADB 表现出色。此外，我们将 LADB 扩展至处理多源翻译（来自深度图和分割掩码）以及在条件样式迁移任务中的多目标翻译，展示了其在处理多样且异构应用场景时的灵活性。 最终，我们提出 LADB 作为一个可扩展且灵活的解决方案，用于解决现实世界领域的转换问题，特别是那些数据标注成本高昂或不完整的场景。
URL
https://arxiv.org/abs/2509.08628
PDF
-
OmniStyle2: Scalable and High Quality Artistic Style Transfer Data Generation via Destylization
2025-09-07 08:22:17Ye Wang, Zili Yi, Yibo Zhang, Peng Zheng, Xuping Xie, Jiang Lin, Yilin Wang, Rui MaAbstract
OmniStyle2 introduces a novel approach to artistic style transfer by reframing it as a data problem. Our key insight is destylization, reversing style transfer by removing stylistic elements from artworks to recover natural, style-free counterparts. This yields DST-100K, a large-scale dataset that provides authentic supervision signals by aligning real artistic styles with their underlying content. To build DST-100K, we develop (1) DST, a text-guided destylization model that reconstructs stylefree content, and (2) DST-Filter, a multi-stage evaluation model that employs Chain-of-Thought reasoning to automatically discard low-quality pairs while ensuring content fidelity and style accuracy. Leveraging DST-100K, we train OmniStyle2, a simple feed-forward model based on FLUX.1-dev. Despite its simplicity, OmniStyle2 consistently surpasses state-of-the-art methods across both qualitative and quantitative benchmarks. Our results demonstrate that scalable data generation via destylization provides a reliable supervision paradigm, overcoming the fundamental challenge posed by the lack of ground-truth data in artistic style transfer.
Abstract (translated)
OmniStyle2 引入了一种新颖的艺术风格转换方法，将这一过程重新定义为一个数据问题。我们的关键洞察是去风格化（destylization），即通过移除艺术品中的风格元素来逆转风格转移，从而恢复自然且无特定风格的原作。这种方法产生了 DST-100K 数据集，这是一个大规模的数据集，它提供真实的监督信号，将实际的艺术风格与其底层内容相匹配。 为了构建 DST-100K，我们开发了以下两个模型： 1. DST：一个文本引导去风格化（destylization）模型，用于重建无特定风格的内容。 2. DST-Filter：一个多阶段评估模型，采用链式思维推理（Chain-of-Thought reasoning），自动筛选出低质量的样本对，并确保内容的真实性和风格的准确性。 利用 DST-100K 数据集，我们训练了 OmniStyle2，这是一个基于 FLUX.1-dev 的简单前馈模型。尽管其结构相对简单，OmniStyle2 在定性与定量基准测试中始终优于现有的最先进方法。 我们的研究结果表明，通过去风格化生成可扩展的数据提供了可靠的监督范例，从而克服了艺术风格转移中由于缺乏真实数据所面临的根本挑战。
URL
https://arxiv.org/abs/2509.05970
PDF
-
SSGaussian: Semantic-Aware and Structure-Preserving 3D Style Transfer
2025-09-04 16:40:44Jimin Xu, Bosheng Qin, Tao Jin, Zhou Zhao, Zhenhui Ye, Jun Yu, Fei WuAbstract
Recent advancements in neural representations, such as Neural Radiance Fields and 3D Gaussian Splatting, have increased interest in applying style transfer to 3D scenes. While existing methods can transfer style patterns onto 3D-consistent neural representations, they struggle to effectively extract and transfer high-level style semantics from the reference style image. Additionally, the stylized results often lack structural clarity and separation, making it difficult to distinguish between different instances or objects within the 3D scene. To address these limitations, we propose a novel 3D style transfer pipeline that effectively integrates prior knowledge from pretrained 2D diffusion models. Our pipeline consists of two key stages: First, we leverage diffusion priors to generate stylized renderings of key viewpoints. Then, we transfer the stylized key views onto the 3D representation. This process incorporates two innovative designs. The first is cross-view style alignment, which inserts cross-view attention into the last upsampling block of the UNet, allowing feature interactions across multiple key views. This ensures that the diffusion model generates stylized key views that maintain both style fidelity and instance-level consistency. The second is instance-level style transfer, which effectively leverages instance-level consistency across stylized key views and transfers it onto the 3D representation. This results in a more structured, visually coherent, and artistically enriched stylization. Extensive qualitative and quantitative experiments demonstrate that our 3D style transfer pipeline significantly outperforms state-of-the-art methods across a wide range of scenes, from forward-facing to challenging 360-degree environments. Visit our project page this https URL for immersive visualization.
Abstract (translated)
近期在神经表示领域的进展，如神经辐射场（Neural Radiance Fields）和3D高斯点阵（3D Gaussian Splatting），引发了将风格迁移应用于三维场景的兴趣。尽管现有的方法能够将样式图案转移到具有三维一致性的神经表示上，但它们难以有效提取并转移参考样式图像中的高级别样式语义信息。此外，经过样式的处理结果往往缺乏结构清晰度和分离性，使得在三维场景中区分不同的实例或物体变得困难。 为了解决这些问题，我们提出了一种新的3D风格迁移流水线，该管道能够有效地整合预训练的2D扩散模型中的先验知识。我们的流水线包含两个关键阶段：首先，利用扩散先验生成主要视角下的样式化渲染；然后将这些样式化的主视图转移到三维表示上。这一过程包含了两项创新设计。 第一项是跨视角风格对齐（Cross-view style alignment），它在U-Net的最后一个上采样块中插入了跨视角注意力机制，允许多个关键视点之间的特征交互。这确保扩散模型生成的样式化主视图既保持样式保真度又维护实例级别的连贯性。 第二项是实例级风格迁移（Instance-level style transfer），它有效地利用了不同样式化主视图之间的一致性，并将其转移至三维表示上，从而产生结构更清晰、视觉一致性更强且艺术效果更加丰富的样化处理结果。 通过广泛的定性和定量实验证明，我们的3D风格迁移流水线在从正面对准到挑战性的全方位环境等各种场景中都显著优于当前最先进的方法。欢迎访问我们项目的网页 [这个链接]（请将此链接替换为实际项目页面的URL）以进行沉浸式可视化体验。
URL
https://arxiv.org/abs/2509.04379
PDF
-
Improved 3D Scene Stylization via Text-Guided Generative Image Editing with Region-Based Control
2025-09-04 15:01:01Haruo Fujiwara, Yusuke Mukuta, Tatsuya HaradaAbstract
Recent advances in text-driven 3D scene editing and stylization, which leverage the powerful capabilities of 2D generative models, have demonstrated promising outcomes. However, challenges remain in ensuring high-quality stylization and view consistency simultaneously. Moreover, applying style consistently to different regions or objects in the scene with semantic correspondence is a challenging task. To address these limitations, we introduce techniques that enhance the quality of 3D stylization while maintaining view consistency and providing optional region-controlled style transfer. Our method achieves stylization by re-training an initial 3D representation using stylized multi-view 2D images of the source views. Therefore, ensuring both style consistency and view consistency of stylized multi-view images is crucial. We achieve this by extending the style-aligned depth-conditioned view generation framework, replacing the fully shared attention mechanism with a single reference-based attention-sharing mechanism, which effectively aligns style across different viewpoints. Additionally, inspired by recent 3D inpainting methods, we utilize a grid of multiple depth maps as a single-image reference to further strengthen view consistency among stylized images. Finally, we propose Multi-Region Importance-Weighted Sliced Wasserstein Distance Loss, allowing styles to be applied to distinct image regions using segmentation masks from off-the-shelf models. We demonstrate that this optional feature enhances the faithfulness of style transfer and enables the mixing of different styles across distinct regions of the scene. Experimental evaluations, both qualitative and quantitative, demonstrate that our pipeline effectively improves the results of text-driven 3D stylization.
Abstract (translated)
近期在文本驱动的三维场景编辑和风格化方面取得的进步，利用了二维生成模型的强大能力，并展示了有希望的结果。然而，在确保高质量的风格化的同时保持视角一致性仍然存在挑战。此外，将样式一致地应用于具有语义对应的不同区域或物体是一项艰巨的任务。为了解决这些限制，我们引入了一系列技术来增强3D风格化的质量，同时保持视角一致性并提供可选的区域控制式样传输。我们的方法通过使用源视图的风格化多视角2D图像重新训练初始3D表示来实现样式化。因此，确保风格化多视角图像的风格一致性和视角一致性至关重要。 为了达到这一目标，我们扩展了与风格对齐的深度条件视角生成框架，并用单一参考基础注意力共享机制替换了完全共享注意力机制，这有效地将不同视点上的风格进行了对齐。此外，受最近的3D修复方法启发，我们利用多个深度图网格作为单张图像的参考，进一步加强了风格化图像之间的视角一致性。 最后，我们提出了一种多区域重要性加权切片Wasserstein距离损失，通过使用现成模型生成的分割掩码，允许不同的样式应用于独特的图像区域。证明这一可选特性增强了样式的传输保真度，并使不同区域混合不同样式成为可能。实验评估（定性和定量）表明，我们的流程有效地提高了文本驱动3D风格化的结果。 翻译总结了原文献的主要贡献和创新点，展示了如何通过一系列技术改进来解决当前三维场景编辑和风格化中的主要挑战，从而提升图像的质量、一致性和灵活性。
URL
https://arxiv.org/abs/2509.05285
PDF
-
SMooGPT: Stylized Motion Generation using Large Language Models
2025-09-04 09:41:18Lei Zhong, Yi Yang, Changjian LiAbstract
Stylized motion generation is actively studied in computer graphics, especially benefiting from the rapid advances in diffusion models. The goal of this task is to produce a novel motion respecting both the motion content and the desired motion style, e.g., ``walking in a loop like a Monkey''. Existing research attempts to address this problem via motion style transfer or conditional motion generation. They typically embed the motion style into a latent space and guide the motion implicitly in a latent space as well. Despite the progress, their methods suffer from low interpretability and control, limited generalization to new styles, and fail to produce motions other than ``walking'' due to the strong bias in the public stylization dataset. In this paper, we propose to solve the stylized motion generation problem from a new perspective of reasoning-composition-generation, based on our observations: i) human motion can often be effectively described using natural language in a body-part centric manner, ii) LLMs exhibit a strong ability to understand and reason about human motion, and iii) human motion has an inherently compositional nature, facilitating the new motion content or style generation via effective recomposing. We thus propose utilizing body-part text space as an intermediate representation, and present SMooGPT, a fine-tuned LLM, acting as a reasoner, composer, and generator when generating the desired stylized motion. Our method executes in the body-part text space with much higher interpretability, enabling fine-grained motion control, effectively resolving potential conflicts between motion content and style, and generalizes well to new styles thanks to the open-vocabulary ability of LLMs. Comprehensive experiments and evaluations, and a user perceptual study, demonstrate the effectiveness of our approach, especially under the pure text-driven stylized motion generation.
Abstract (translated)
在计算机图形学领域，风格化运动生成是一个活跃的研究方向，尤其是得益于扩散模型的快速进步。该任务的目标是产生一种新颖的运动模式，同时尊重原有的动作内容和所需的动作风格，例如“像猴子一样循环行走”。现有研究试图通过动作样式转换或条件动作生成来解决这个问题。它们通常将动作风格嵌入到潜在空间中，并在相同的潜在空间中隐式引导动作。尽管取得了进展，但这些方法仍然存在解释性差、控制力弱以及无法泛化至新样式的限制，由于公共样例数据集中强烈的偏差问题，导致除了“行走”之外的动作难以生成。 在这篇论文中，我们提出了从推理-组合-生成的新视角来解决风格化运动生成的问题。基于我们的观察：i) 人类动作通常可以通过身体部位为中心的自然语言有效描述；ii) 大型语言模型（LLMs）在理解和推断人类动作方面表现出强大的能力；iii) 人类动作具有内在的组合特性，有助于通过有效的重组来生成新的动作内容或风格。因此，我们建议利用身体部位文本空间作为中间表示，并提出SMooGPT——一个经过微调的大型语言模型，在生成所需的样式化运动时充当推理者、组合器和生成器的角色。我们的方法在人体部位文本空间中执行，具有更高的解释性，支持精细的动作控制，有效解决动作内容与风格之间的潜在冲突，并且由于大型语言模型的开放词汇能力而很好地泛化到新样式。 综合实验、评估以及用户感知研究证明了我们方法的有效性，特别是在纯文本驱动的风格化运动生成方面的优势。
URL
https://arxiv.org/abs/2509.04058
PDF
-
Generative AI for Enhanced Wildfire Detection: Bridging the Synthetic-Real Domain Gap
2025-09-02 00:40:55Satyam GabaAbstract
The early detection of wildfires is a critical environmental challenge, with timely identification of smoke plumes being key to mitigating large-scale damage. While deep neural networks have proven highly effective for localization tasks, the scarcity of large, annotated datasets for smoke detection limits their potential. In response, we leverage generative AI techniques to address this data limitation by synthesizing a comprehensive, annotated smoke dataset. We then explore unsupervised domain adaptation methods for smoke plume segmentation, analyzing their effectiveness in closing the gap between synthetic and real-world data. To further refine performance, we integrate advanced generative approaches such as style transfer, Generative Adversarial Networks (GANs), and image matting. These methods aim to enhance the realism of synthetic data and bridge the domain disparity, paving the way for more accurate and scalable wildfire detection models.
Abstract (translated)
URL
https://arxiv.org/abs/2511.16617
PDF
-
PRINTER:Deformation-Aware Adversarial Learning for Virtual IHC Staining with In Situ Fidelity
2025-09-01 07:53:05Yizhe Yuan, Bingsen Xue, Bangzheng Pu, Chengxiang Wang, Cheng JinAbstract
Tumor spatial heterogeneity analysis requires precise correlation between Hematoxylin and Eosin H&E morphology and immunohistochemical (IHC) biomarker expression, yet current methods suffer from spatial misalignment in consecutive sections, severely compromising in situ pathological interpretation. In order to obtain a more accurate virtual staining pattern, We propose PRINTER, a weakly-supervised framework that integrates PRototype-drIven content and staiNing patTERn decoupling and deformation-aware adversarial learning strategies designed to accurately learn IHC staining patterns while preserving H&E staining details. Our approach introduces three key innovations: (1) A prototype-driven staining pattern transfer with explicit content-style decoupling; and (2) A cyclic registration-synthesis framework GapBridge that bridges H&E and IHC domains through deformable structural alignment, where registered features guide cross-modal style transfer while synthesized outputs iteratively refine the registration;(3) Deformation-Aware Adversarial Learning: We propose a training framework where a generator and deformation-aware registration network jointly adversarially optimize a style-focused discriminator. Extensive experiments demonstrate that PRINTER effectively achieves superior performance in preserving H&E staining details and virtual staining fidelity, outperforming state-of-the-art methods. Our work provides a robust and scalable solution for virtual staining, advancing the field of computational pathology.
Abstract (translated)
肿瘤空间异质性分析需要精确地将苏木精和伊红（H&E）形态学与免疫组化（IHC）生物标志物表达相关联，然而当前的方法因连续切片中的空间对齐问题而遭受重大影响，这严重削弱了原位病理解释的准确性。为了获得更准确的虚拟染色模式，我们提出了一种弱监督框架——PRINTER，该框架集成了原型驱动的内容和染色模式解耦以及感知变形对抗性学习策略的设计，旨在在保持H&E染色细节的同时精确地学习IHC染色模式。我们的方法引入了三个关键创新： 1. 原型驱动的染色模式转移，并明确内容-风格分离； 2. 一个循环注册-合成框架GapBridge，通过可变形结构对齐连接H&E和IHC领域，在此过程中，已注册特征指导跨模态风格转换而合成输出则迭代优化注册过程； 3. 感知变形的对抗性学习：我们提出了一种训练框架，在该框架中生成器与感知变形的配准网络联合对抗地优化以焦点风格为中心的判别器。 广泛的实验表明，PRINTER在保留H&E染色细节和虚拟染色保真度方面表现出了卓越性能，并且优于最先进的方法。我们的工作为虚拟染色提供了一个稳健且可扩展的解决方案，推动了计算病理学领域的发展。
URL
https://arxiv.org/abs/2509.01214
PDF
-
TRUST: Token-dRiven Ultrasound Style Transfer for Cross-Device Adaptation
2025-08-30 14:00:50Nhat-Tuong Do-Tran, Ngoc-Hoang-Lam Le, Ian Chiu, Po-Tsun Paul Kuo, Ching-Chun HuangAbstract
Ultrasound images acquired from different devices exhibit diverse styles, resulting in decreased performance of downstream tasks. To mitigate the style gap, unpaired image-to-image (UI2I) translation methods aim to transfer images from a source domain, corresponding to new device acquisitions, to a target domain where a frozen task model has been trained for downstream applications. However, existing UI2I methods have not explicitly considered filtering the most relevant style features, which may result in translated images misaligned with the needs of downstream tasks. In this work, we propose TRUST, a token-driven dual-stream framework that preserves source content while transferring the common style of the target domain, ensuring that content and style remain unblended. Given multiple styles in the target domain, we introduce a Token-dRiven (TR) module that operates from two perspectives: (1) a data view--selecting "suitable" target tokens corresponding to each source token, and (2) a model view--identifying ``optimal" target tokens for the downstream model, guided by a behavior mirror loss. Additionally, we inject auxiliary prompts into the source encoder to match content representation with downstream behavior. Experimental results on ultrasound datasets demonstrate that TRUST outperforms existing UI2I methods in both visual quality and downstream task performance.
Abstract (translated)
不同设备获取的超声图像风格各异，这会导致下游任务性能下降。为了缩小这种风格差距，未配对图像到图像（UI2I）转换方法旨在将源领域中的图像（即新设备采集的图像）转换为目标领域的图像，在目标领域中已经为下游应用训练好了冻结的任务模型。然而，现有的UI2I方法没有明确地考虑筛选最相关的风格特征，这可能导致转化后的图像与下游任务的需求不一致。 为此，我们提出了TRUST——一种基于令牌驱动的双流框架，该框架在保留源内容的同时转移目标领域的通用风格，并确保内容和风格不会混合。面对目标领域中存在多种风格的情况，我们引入了Token-dRiven（TR）模块，从两个视角进行操作：(1) 数据视角——为每个源标记选择相应的“合适”的目标标记；(2) 模型视角——识别对于下游模型来说的“最佳”目标标记，这由行为镜像损失指导完成。此外，我们还在源编码器中注入了辅助提示，以使内容表示与下游行为相匹配。 在超声数据集上的实验结果表明，TRUST不仅在视觉质量方面优于现有的UI2I方法，在下游任务性能上也同样表现出色。
URL
https://arxiv.org/abs/2509.00508
PDF
-
Print2Volume: Generating Synthetic OCT-based 3D Fingerprint Volume from 2D Fingerprint Image
2025-08-29 07:26:39Qingran Miao, Haixia Wang, Haohao Sun, Yilong ZhangAbstract
Optical Coherence Tomography (OCT) enables the acquisition of high-resolution, three-dimensional fingerprint data, capturing rich subsurface structures for robust biometric recognition. However, the high cost and time-consuming nature of OCT data acquisition have led to a scarcity of large-scale public datasets, significantly hindering the development of advanced algorithms, particularly data-hungry deep learning models. To address this critical bottleneck, this paper introduces Print2Volume, a novel framework for generating realistic, synthetic OCT-based 3D fingerprints from 2D fingerprint image. Our framework operates in three sequential stages: (1) a 2D style transfer module that converts a binary fingerprint into a grayscale images mimicking the style of a Z-direction mean-projected OCT scan; (2) a 3D Structure Expansion Network that extrapolates the 2D im-age into a plausible 3D anatomical volume; and (3) an OCT Realism Refiner, based on a 3D GAN, that renders the structural volume with authentic textures, speckle noise, and other imaging characteristics. Using Print2Volume, we generated a large-scale synthetic dataset of 420,000 samples. Quantitative experiments demonstrate the high quality of our synthetic data and its significant impact on recognition performance. By pre-training a recognition model on our synthetic data and fine-tuning it on a small real-world dataset, we achieved a remarkable reduction in the Equal Error Rate (EER) from 15.62% to 2.50% on the ZJUT-EIFD benchmark, proving the effectiveness of our approach in overcoming data scarcity.
Abstract (translated)
光学相干断层扫描（OCT）能够获取高分辨率的三维指纹数据，捕捉丰富的亚表面结构，从而实现稳健的生物识别。然而，由于OCT数据采集成本高昂且耗时较长，导致大型公共数据集稀缺，严重阻碍了先进算法尤其是对大量数据有需求的深度学习模型的发展。 为了解决这一关键瓶颈问题，本文提出了Print2Volume框架，该框架可以从二维指纹图像生成真实的基于OCT的三维合成指纹。我们的框架包括三个连续阶段：（1）一个二维风格转换模块，将二值化的指纹图转化为模拟Z方向均值投影的OCT扫描风格的灰度图像；（2）一个3D结构扩展网络，该网络从2D图像推断出合理的真实生物体体积；以及（3）基于三维生成对抗网络（GAN）的OCT现实性增强器，用于以真实的纹理、斑点噪声和其他成像特征来渲染解剖结构。 利用Print2Volume框架，我们创建了一个包含420,000个样本的大规模合成数据集。定量实验显示了我们的合成数据的质量及其对识别性能的重要影响。通过在我们合成的数据上预训练一个识别模型，并使用小规模的真实世界数据进行微调，我们在ZJUT-EIFD基准测试中将等错误率（EER）从15.62%显著降低到了2.50%，证明了我们的方法在克服数据稀缺性方面的有效性。
URL
https://arxiv.org/abs/2508.21371
PDF
-
CraftGraffiti: Exploring Human Identity with Custom Graffiti Art via Facial-Preserving Diffusion Models
2025-08-28 10:38:13Ayan Banerjee, Fernando Vilari\~no, Josep Llad\'osAbstract
Preserving facial identity under extreme stylistic transformation remains a major challenge in generative art. In graffiti, a high-contrast, abstract medium, subtle distortions to the eyes, nose, or mouth can erase the subject's recognizability, undermining both personal and cultural authenticity. We present CraftGraffiti, an end-to-end text-guided graffiti generation framework designed with facial feature preservation as a primary objective. Given an input image and a style and pose descriptive prompt, CraftGraffiti first applies graffiti style transfer via LoRA-fine-tuned pretrained diffusion transformer, then enforces identity fidelity through a face-consistent self-attention mechanism that augments attention layers with explicit identity embeddings. Pose customization is achieved without keypoints, using CLIP-guided prompt extension to enable dynamic re-posing while retaining facial coherence. We formally justify and empirically validate the "style-first, identity-after" paradigm, showing it reduces attribute drift compared to the reverse order. Quantitative results demonstrate competitive facial feature consistency and state-of-the-art aesthetic and human preference scores, while qualitative analyses and a live deployment at the Cruilla Festival highlight the system's real-world creative impact. CraftGraffiti advances the goal of identity-respectful AI-assisted artistry, offering a principled approach for blending stylistic freedom with recognizability in creative AI applications.
Abstract (translated)
在极端风格变换中保持面部身份识别是一个生成艺术中的重大挑战。在涂鸦这种高对比度、抽象的艺术形式中，对眼睛、鼻子或嘴巴的细微扭曲可能会抹去主体的可辨识性，从而削弱个人和文化的独特性。我们推出了CraftGraffiti，这是一个端到端的文本引导式涂鸦生成框架，旨在以面部特征保真为主要目标。 给定输入图像以及描述风格与姿态的文字提示后，CraftGraffiti首先通过使用LoRA微调预训练扩散变换器来应用涂鸦风格转移。然后，它通过一个增强注意力层并添加显式身份嵌入的面一致自我注意机制来确保身份保真度。姿势定制在无需关键点的情况下完成，通过CLIP引导的文字提示扩展实现动态重新定位的同时保持面部一致性。 我们正式证明并在实证上验证了“风格优先、身份随后”的范例，表明与相反顺序相比，这种方法可以减少属性漂移（attribute drift）。定量结果显示了具有竞争力的面部特征一致性和美学以及人类偏好的最佳评分。定性分析和在Cruilla音乐节上的实际部署突显了该系统的现实世界创意影响力。 CraftGraffiti推进了尊重身份的AI辅助艺术的目标，提供了一种将风格自由度与可辨识性相结合的原则方法，在创造性的AI应用中开辟新的领域。
URL
https://arxiv.org/abs/2508.20640
PDF
-
PersonaAnimator: Personalized Motion Transfer from Unconstrained Videos
2025-08-27 13:52:25Ziyun Qian, Runyu Xiao, Shuyuan Tu, Wei Xue, Dingkang Yang, Mingcheng Li, Dongliang Kou, Minghao Han, Zizhi Chen, Lihua ZhangAbstract
Recent advances in motion generation show remarkable progress. However, several limitations remain: (1) Existing pose-guided character motion transfer methods merely replicate motion without learning its style characteristics, resulting in inexpressive characters. (2) Motion style transfer methods rely heavily on motion capture data, which is difficult to obtain. (3) Generated motions sometimes violate physical laws. To address these challenges, this paper pioneers a new task: Video-to-Video Motion Personalization. We propose a novel framework, PersonaAnimator, which learns personalized motion patterns directly from unconstrained videos. This enables personalized motion transfer. To support this task, we introduce PersonaVid, the first video-based personalized motion dataset. It contains 20 motion content categories and 120 motion style categories. We further propose a Physics-aware Motion Style Regularization mechanism to enforce physical plausibility in the generated motions. Extensive experiments show that PersonaAnimator outperforms state-of-the-art motion transfer methods and sets a new benchmark for the Video-to-Video Motion Personalization task.
Abstract (translated)
最近的运动生成技术取得了显著进展，但仍然存在一些局限性：（1）现有的姿态引导的角色运动转移方法仅复制动作而不学习其风格特征，导致角色表现力不足。（2）运动风格转移方法严重依赖于难以获取的动作捕捉数据。（3）生成的运动有时会违反物理定律。为解决这些挑战，本文提出了一项新任务：视频到视频的个性化运动（Video-to-Video Motion Personalization）。我们提出了一个创新框架PersonaAnimator，该框架可以直接从无约束视频中学习个性化的运动模式，从而实现个性化的运动转移。为了支持这项任务，我们引入了PersonaVid，这是第一个基于视频的个性化动作数据集。它包含了20种不同的运动内容类别和120种不同的运动风格类别。此外，我们还提出了一种物理感知的动作风格正则化机制，以确保生成的动作符合物理可能性。大量的实验表明，PersonaAnimator在个性化的运动转移方法中表现优于现有的最新技术，并为视频到视频的个性化运动任务设立了新的基准。
URL
https://arxiv.org/abs/2508.19895
PDF
-
CompLex: Music Theory Lexicon Constructed by Autonomous Agents for Automatic Music Generation
2025-08-27 06:36:49Zhejing Hu, Yan Liu, Gong Chen, Bruce X. B. YuAbstract
Generative artificial intelligence in music has made significant strides, yet it still falls short of the substantial achievements seen in natural language processing, primarily due to the limited availability of music data. Knowledge-informed approaches have been shown to enhance the performance of music generation models, even when only a few pieces of musical knowledge are integrated. This paper seeks to leverage comprehensive music theory in AI-driven music generation tasks, such as algorithmic composition and style transfer, which traditionally require significant manual effort with existing techniques. We introduce a novel automatic music lexicon construction model that generates a lexicon, named CompLex, comprising 37,432 items derived from just 9 manually input category keywords and 5 sentence prompt templates. A new multi-agent algorithm is proposed to automatically detect and mitigate hallucinations. CompLex demonstrates impressive performance improvements across three state-of-the-art text-to-music generation models, encompassing both symbolic and audio-based methods. Furthermore, we evaluate CompLex in terms of completeness, accuracy, non-redundancy, and executability, confirming that it possesses the key characteristics of an effective lexicon.
Abstract (translated)
音乐领域的生成式人工智能已经取得了显著的进步，但仍然不及自然语言处理领域取得的重大成就，这主要是由于可用的音乐数据有限。知识驱动的方法已被证明可以提高音乐生成模型的表现，即使只是整合了少量的音乐知识也是如此。本文旨在通过利用全面的音乐理论来增强AI驱动的音乐创作任务（如算法作曲和风格转换），这些任务通常需要现有技术下大量的手动工作才能完成。我们提出了一种新的自动构建音乐词汇表的模型，该模型生成了一个名为CompLex的词汇库，包含由9个手动输入类别关键词和5个句子提示模板产生的37,432个项目。此外，还提出了一个新的多代理算法来自动检测并减轻幻觉问题。在三个最先进的文本到音乐生成模型上进行测试时，CompLex显示了显著的表现提升，这些模型涵盖了符号基和音频基方法。我们从完整度、准确性、非冗余性和可执行性等方面评估了CompLex，确认其具备了一个有效词汇表的关键特性。
URL
https://arxiv.org/abs/2508.19603
PDF
