Style_Transfer
-
SGSST: Scaling Gaussian Splatting StyleTransfer
2024-12-04 14:59:05Bruno Galerne, Jianling Wang, Lara Raad, Jean-Michel MorelAbstract
Applying style transfer to a full 3D environment is a challenging task that has seen many developments since the advent of neural rendering. 3D Gaussian splatting (3DGS) has recently pushed further many limits of neural rendering in terms of training speed and reconstruction quality. This work introduces SGSST: Scaling Gaussian Splatting Style Transfer, an optimization-based method to apply style transfer to pretrained 3DGS scenes. We demonstrate that a new multiscale loss based on global neural statistics, that we name SOS for Simultaneously Optimized Scales, enables style transfer to ultra-high resolution 3D scenes. Not only SGSST pioneers 3D scene style transfer at such high image resolutions, it also produces superior visual quality as assessed by thorough qualitative, quantitative and perceptual comparisons.
Abstract (translated)
URL
https://arxiv.org/abs/2412.03371
PDF
-
The Role of Text-to-Image Models in Advanced Style Transfer Applications: A Case Study with DALL-E 3
Abstract
While DALL-E 3 has gained popularity for its ability to generate creative and complex images from textual descriptions, its application in the domain of style transfer remains slightly underexplored. This project investigates the integration of DALL-E 3 with traditional neural style transfer techniques to assess the impact of generated style images on the quality of the final output. DALL-E 3 was employed to generate style images based on the descriptions provided and combine these with the Magenta Arbitrary Image Stylization model. This integration is evaluated through metrics such as the Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR), as well as processing time assessments. The findings reveal that DALL-E 3 significantly enhances the diversity and artistic quality of stylized images. Although this improvement comes with a slight increase in style transfer time, the data shows that this trade-off is worthwhile because the overall processing time with DALL-E 3 is about 2.5 seconds faster than traditional methods, making it both an efficient and visually superior option.
Abstract (translated)
尽管DALL-E 3因其能够根据文本描述生成创意和复杂的图像而受到欢迎，但在风格迁移领域的应用仍稍显不足。本项目研究了将DALL-E 3与传统的神经风格迁移技术相结合，以评估生成的风格图像对最终输出质量的影响。使用DALL-E 3基于提供的描述生成风格图像，并将其与Magenta任意图像样式化模型结合。这种整合通过结构相似性指数测量（SSIM）和峰值信噪比（PSNR）等指标以及处理时间评估来进行评价。研究结果表明，DALL-E 3显著增强了风格化图像的多样性和艺术质量。虽然这种方法在风格迁移时会稍微增加一些时间，但数据表明这种权衡是值得的，因为使用DALL-E 3的整体处理时间比传统方法快约2.5秒，使其成为既高效又视觉上更优的选择。
URL
https://arxiv.org/abs/2412.05325
PDF
-
GIST: Towards Photorealistic Style Transfer via Multiscale Geometric Representations
2024-12-03 07:05:39Renan A. Rojas-Gomez, Minh N. DoAbstract
State-of-the-art Style Transfer methods often leverage pre-trained encoders optimized for discriminative tasks, which may not be ideal for image synthesis. This can result in significant artifacts and loss of photorealism. Motivated by the ability of multiscale geometric image representations to capture fine-grained details and global structure, we propose GIST: Geometric-based Image Style Transfer, a novel Style Transfer technique that exploits the geometric properties of content and style images. GIST replaces the standard Neural Style Transfer autoencoding framework with a multiscale image expansion, preserving scene details without the need for post-processing or training. Our method matches multiresolution and multidirectional representations such as Wavelets and Contourlets by solving an optimal transport problem, leading to an efficient texture transferring. Experiments show that GIST is on-par or outperforms recent photorealistic Style Transfer approaches while significantly reducing the processing time with no model training.
Abstract (translated)
URL
https://arxiv.org/abs/2412.02214
PDF
-
SADG: Segment Any Dynamic Gaussian Without Object Trackers
2024-11-28 17:47:48Yun-Jin Li, Mariia Gladkova, Yan Xia, Daniel CremersAbstract
Understanding dynamic 3D scenes is fundamental for various applications, including extended reality (XR) and autonomous driving. Effectively integrating semantic information into 3D reconstruction enables holistic representation that opens opportunities for immersive and interactive applications. We introduce SADG, Segment Any Dynamic Gaussian Without Object Trackers, a novel approach that combines dynamic Gaussian Splatting representation and semantic information without reliance on object IDs. In contrast to existing works, we do not rely on supervision based on object identities to enable consistent segmentation of dynamic 3D objects. To this end, we propose to learn semantically-aware features by leveraging masks generated from the Segment Anything Model (SAM) and utilizing our novel contrastive learning objective based on hard pixel mining. The learned Gaussian features can be effectively clustered without further post-processing. This enables fast computation for further object-level editing, such as object removal, composition, and style transfer by manipulating the Gaussians in the scene. We further extend several dynamic novel-view datasets with segmentation benchmarks to enable testing of learned feature fields from unseen viewpoints. We evaluate SADG on proposed benchmarks and demonstrate the superior performance of our approach in segmenting objects within dynamic scenes along with its effectiveness for further downstream editing tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2411.19290
PDF
-
Face2QR: A Unified Framework for Aesthetic, Face-Preserving, and Scannable QR Code Generation
2024-11-28 16:35:16Xuehao Cui, Guangyang Wu, Zhenghao Gan, Guangtao Zhai, Xiaohong LiuAbstract
Existing methods to generate aesthetic QR codes, such as image and style transfer techniques, tend to compromise either the visual appeal or the scannability of QR codes when they incorporate human face identity. Addressing these imperfections, we present Face2QR-a novel pipeline specifically designed for generating personalized QR codes that harmoniously blend aesthetics, face identity, and scannability. Our pipeline introduces three innovative components. First, the ID-refined QR integration (IDQR) seamlessly intertwines the background styling with face ID, utilizing a unified Stable Diffusion (SD)-based framework with control networks. Second, the ID-aware QR ReShuffle (IDRS) effectively rectifies the conflicts between face IDs and QR patterns, rearranging QR modules to maintain the integrity of facial features without compromising scannability. Lastly, the ID-preserved Scannability Enhancement (IDSE) markedly boosts scanning robustness through latent code optimization, striking a delicate balance between face ID, aesthetic quality and QR functionality. In comprehensive experiments, Face2QR demonstrates remarkable performance, outperforming existing approaches, particularly in preserving facial recognition features within custom QR code designs. Codes are available at $\href{this https URL}{\text{this URL link}}$.
Abstract (translated)
URL
https://arxiv.org/abs/2411.19246
PDF
-
Z-STAR+: A Zero-shot Style Transfer Method via Adjusting Style Distribution
2024-11-28 15:56:17Yingying Deng, Xiangyu He, Fan Tang, Weiming DongAbstract
Style transfer presents a significant challenge, primarily centered on identifying an appropriate style representation. Conventional methods employ style loss, derived from second-order statistics or contrastive learning, to constrain style representation in the stylized result. However, these pre-defined style representations often limit stylistic expression, leading to artifacts. In contrast to existing approaches, we have discovered that latent features in vanilla diffusion models inherently contain natural style and content distributions. This allows for direct extraction of style information and seamless integration of generative priors into the content image without necessitating retraining. Our method adopts dual denoising paths to represent content and style references in latent space, subsequently guiding the content image denoising process with style latent codes. We introduce a Cross-attention Reweighting module that utilizes local content features to query style image information best suited to the input patch, thereby aligning the style distribution of the stylized results with that of the style image. Furthermore, we design a scaled adaptive instance normalization to mitigate inconsistencies in color distribution between style and stylized images on a global scale. Through theoretical analysis and extensive experimentation, we demonstrate the effectiveness and superiority of our diffusion-based \uline{z}ero-shot \uline{s}tyle \uline{t}ransfer via \uline{a}djusting style dist\uline{r}ibution, termed Z-STAR+.
Abstract (translated)
URL
https://arxiv.org/abs/2411.19231
PDF
-
Music2Fail: Transfer Music to Failed Recorder Style
2024-11-27 06:05:29Chon In Leong, I-Ling Chung, Kin-Fong Chao, Jun-You Wang, Yi-Hsuan Yang, Jyh-Shing Roger JangAbstract
The goal of music style transfer is to convert a music performance by one instrument into another while keeping the musical contents unchanged. In this paper, we investigate another style transfer scenario called ``failed-music style transfer''. Unlike the usual music style transfer where the content remains the same and only the instrumental characteristics are changed, this scenario seeks to transfer the music from the source instrument to the target instrument which is deliberately performed off-pitch. Our work attempts to transfer normally played music into off-pitch recorder music, which we call ``failed-style recorder'', and study the results of the conversion. To carry out this work, we have also proposed a dataset of failed-style recorders for this task, called ``FR109 Dataset''. Such an experiment explores the music style transfer task in a more expressive setting, as the generated audio should sound like an ``off-pitch recorder'' while maintaining a certain degree of naturalness.
Abstract (translated)
音乐风格转换的目标是将一种乐器的演奏转换为另一种，同时保持音乐内容不变。在本文中，我们研究了另一种称为“失败音乐风格转换”的场景。与通常的音乐风格转换不同，在这种情况下内容保持不变，只有乐器特性发生变化，我们的目标是在此场景下将原乐器上的音乐转移到故意跑调的目标乐器上。我们的工作尝试将正常演奏的音乐转化为跑调的笛子音乐，我们称之为“失败风格笛子”，并研究转化结果。为此，我们也为该任务提出了一个名为“FR109数据集”的失败风格笛子的数据集。此类实验在更为丰富的表达设置下探索了音乐风格转换的任务，因为生成的音频既要听起来像“跑调的笛子”，同时还要保持一定程度上的自然度。
URL
https://arxiv.org/abs/2411.18075
PDF
-
Interleaved Scene Graph for Interleaved Text-and-Image Generation Assessment
2024-11-26 07:55:57Dongping Chen, Ruoxi Chen, Shu Pu, Zhaoyi Liu, Yanru Wu, Caixi Chen, Benlin Liu, Yue Huang, Yao Wan, Pan Zhou, Ranjay KrishnaAbstract
Many real-world user queries (e.g. "How do to make egg fried rice?") could benefit from systems capable of generating responses with both textual steps with accompanying images, similar to a cookbook. Models designed to generate interleaved text and images face challenges in ensuring consistency within and across these modalities. To address these challenges, we present ISG, a comprehensive evaluation framework for interleaved text-and-image generation. ISG leverages a scene graph structure to capture relationships between text and image blocks, evaluating responses on four levels of granularity: holistic, structural, block-level, and image-specific. This multi-tiered evaluation allows for a nuanced assessment of consistency, coherence, and accuracy, and provides interpretable question-answer feedback. In conjunction with ISG, we introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8 categories and 21 subcategories. This benchmark dataset includes complex language-vision dependencies and golden answers to evaluate models effectively on vision-centric tasks such as style transfer, a challenging area for current models. Using ISG-Bench, we demonstrate that recent unified vision-language models perform poorly on generating interleaved content. While compositional approaches that combine separate language and image models show a 111% improvement over unified models at the holistic level, their performance remains suboptimal at both block and image levels. To facilitate future work, we develop ISG-Agent, a baseline agent employing a "plan-execute-refine" pipeline to invoke tools, achieving a 122% performance improvement.
Abstract (translated)
许多实际的用户查询（例如，“如何制作蛋炒饭？”）可以从能够生成带有文本步骤和配套图片的系统中受益，类似于一本食谱。设计用于生成交错文本和图像的模型面临确保这些模态之间一致性的挑战。为了解决这些问题，我们提出了ISG，一个全面评估交错文本与图像生成的框架。ISG利用场景图结构捕捉文本块和图像块之间的关系，在四个层次上进行评估：整体、结构性、区块级和图片特定。这种多层级评价使得可以细致地评估一致性、连贯性和准确性，并提供可解释的问题答案反馈。 结合ISG，我们引入了一个基准测试——ISG-Bench，涵盖了8个类别21个子类别的1,150个样本。该基准数据集包括复杂的语言视觉依赖关系和黄金标准答案，用于有效地评估以视觉为中心的任务（如风格转换）的模型表现，这是当前模型面临的具有挑战性的领域。 使用ISG-Bench，我们展示了最近统一的视觉-语言模型在生成交错内容方面表现不佳。虽然组合方法将单独的语言和图像模型结合起来，在整体层面上比统一模型提高了111%，但它们在区块级和图片级别的性能仍然次优。为了促进未来的工作，我们开发了ISG-Agent，这是一个采用“规划-执行-优化”流程的基础代理，通过调用工具实现了122%的性能提升。
URL
https://arxiv.org/abs/2411.17188
PDF
-
Synthesising Handwritten Music with GANs: A Comprehensive Evaluation of CycleWGAN, ProGAN, and DCGAN
2024-11-25 14:10:43Elona Shatri, Kalikidhar Palavala, George FazekasAbstract
The generation of handwritten music sheets is a crucial step toward enhancing Optical Music Recognition (OMR) systems, which rely on large and diverse datasets for optimal performance. However, handwritten music sheets, often found in archives, present challenges for digitisation due to their fragility, varied handwriting styles, and image quality. This paper addresses the data scarcity problem by applying Generative Adversarial Networks (GANs) to synthesise realistic handwritten music sheets. We provide a comprehensive evaluation of three GAN models - DCGAN, ProGAN, and CycleWGAN - comparing their ability to generate diverse and high-quality handwritten music images. The proposed CycleWGAN model, which enhances style transfer and training stability, significantly outperforms DCGAN and ProGAN in both qualitative and quantitative evaluations. CycleWGAN achieves superior performance, with an FID score of 41.87, an IS of 2.29, and a KID of 0.05, making it a promising solution for improving OMR systems.
Abstract (translated)
URL
https://arxiv.org/abs/2411.16405
PDF
-
CNNs for Style Transfer of Digital to Film Photography
2024-11-24 19:44:47Pierre Mackenzie, Mika Senghaas, Raphael AchddouAbstract
The use of deep learning in stylistic effect generation has seen increasing use over recent years. In this work, we use simple convolutional neural networks to model Cinestill800T film given a digital input. We test the effect of different loss functions, the addition of an input noise channel and the use of random scales of patches during training. We find that a combination of MSE/VGG loss gives the best colour production and that some grain can be produced, but it is not of a high quality, and no halation is produced. We contribute our dataset of aligned paired images taken with a film and digital camera for further work.
Abstract (translated)
近年来，深度学习在风格效果生成中的应用越来越广泛。在这项工作中，我们使用简单的卷积神经网络根据数字输入来模拟Cinestill800T胶片的效果。我们测试了不同损失函数的影响、添加输入噪声通道以及训练过程中随机缩放图像块的作用。我们发现，结合MSE/VGG损失可以产生最佳的颜色效果，并且能够生成一些颗粒感，但质量不高，也没有生成晕影效果。我们贡献了一组使用胶片相机和数字相机拍摄的对齐配对图像数据集，以供进一步研究。
URL
https://arxiv.org/abs/2411.15967
PDF
-
A Training-Free Approach for Music Style Transfer with Latent Diffusion Models
2024-11-24 16:53:34Sooyoung Kim, Joonwoo Kwon, Heehwan Wang, Shinjae Yoo, Yuewei Lin, Jiook ChaAbstract
Music style transfer, while offering exciting possibilities for personalized music generation, often requires extensive training or detailed textual descriptions. This paper introduces a novel training-free approach leveraging pre-trained Latent Diffusion Models (LDMs). By manipulating the self-attention features of the LDM, we effectively transfer the style of reference music onto content music without additional training. Our method achieves superior style transfer and melody preservation compared to existing methods. This work opens new creative avenues for personalized music generation.
Abstract (translated)
音乐风格转换虽然为个性化音乐生成提供了令人兴奋的可能性，但通常需要大量的训练或详细的文本描述。本文介绍了一种新颖的无需训练的方法，利用预训练的潜在扩散模型（LDM）。通过操纵LDM的自注意力特征，我们能够在不需要额外训练的情况下将参考音乐的风格转移到内容音乐上。与现有方法相比，我们的方法在风格转换和旋律保真方面表现更优。这项工作为个性化音乐生成开辟了新的创意途径。
URL
https://arxiv.org/abs/2411.15913
PDF
-
Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings
2024-11-22 00:09:49Pura Peetathawatchai, Wei-Ning Chen, Berivan Isik, Sanmi Koyejo, Albert NoAbstract
We introduce novel methods for adapting diffusion models under differential privacy (DP) constraints, enabling privacy-preserving style and content transfer without fine-tuning. Traditional approaches to private adaptation, such as DP-SGD, incur significant computational overhead and degrade model performance when applied to large, complex models. Our approach instead leverages embedding-based techniques: Universal Guidance and Textual Inversion (TI), adapted with differentially private mechanisms. We apply these methods to Stable Diffusion for style adaptation using two private datasets: a collection of artworks by a single artist and pictograms from the Paris 2024 Olympics. Experimental results show that the TI-based adaptation achieves superior fidelity in style transfer, even under strong privacy guarantees, while both methods maintain high privacy resilience by employing calibrated noise and subsampling strategies. Our findings demonstrate a feasible and efficient pathway for privacy-preserving diffusion model adaptation, balancing data protection with the fidelity of generated images, and offer insights into embedding-driven methods for DP in generative AI applications.
Abstract (translated)
我们介绍了一种在差分隐私（DP）约束下适应扩散模型的新方法，该方法能够实现在无需微调的情况下进行保护隐私的风格和内容迁移。传统的私有化适应方法如DP-SGD，在应用于大型复杂模型时会导致显著的计算开销并降低模型性能。而我们的方法则利用基于嵌入的技术：通用指导和文本反转（TI），并通过差分隐私机制对其进行调整。我们将这些方法应用于Stable Diffusion，使用两个私有数据集进行风格适应实验：一位艺术家的作品集合以及2024年巴黎奥运会的图标。实验结果表明，在强有力的隐私保证下，基于TI的方法在风格迁移中达到了更高的保真度；同时，两种方法都通过校准噪声和子采样策略保持了较高的隐私韧性。我们的研究发现提供了一条可行且高效的途径，实现了扩散模型适应过程中的数据保护与生成图像保真度之间的平衡，并为嵌入驱动的差分隐私技术在生成式AI应用中提供了见解。
URL
https://arxiv.org/abs/2411.14639
PDF
-
On the Way to LLM Personalization: Learning to Remember User Conversations
2024-11-20 15:45:08Lucie Charlotte Magister, Katherine Metcalf, Yizhe Zhang, Maartje ter HoeveAbstract
Large Language Models (LLMs) have quickly become an invaluable assistant for a variety of tasks. However, their effectiveness is constrained by their ability to tailor responses to human preferences and behaviors via personalization. Prior work in LLM personalization has largely focused on style transfer or incorporating small factoids about the user, as knowledge injection remains an open challenge. In this paper, we explore injecting knowledge of prior conversations into LLMs to enable future work on less redundant, personalized conversations. We identify two real-world constraints: (1) conversations are sequential in time and must be treated as such during training, and (2) per-user personalization is only viable in parameter-efficient settings. To this aim, we propose PLUM, a pipeline performing data augmentation for up-sampling conversations as question-answer pairs, that are then used to finetune a low-rank adaptation adapter with a weighted cross entropy loss. Even in this first exploration of the problem, we perform competitively with baselines such as RAG, attaining an accuracy of 81.5% across 100 conversations.
Abstract (translated)
大型语言模型（LLMs）已迅速成为各种任务中不可或缺的助手。然而，它们的有效性受限于通过个性化调整响应以符合人类偏好和行为的能力。之前关于LLM个性化的研究主要集中在风格转换或融入用户的一些小事实点上，因为知识注入仍然是一个开放挑战。在本文中，我们探索将先前对话的知识注入到LLMs中，以便在未来实现更少冗余且个性化的对话。我们识别了两个现实世界的约束条件：（1）对话是按时间顺序进行的，在训练过程中必须如此处理；（2）面向用户的个性化只有在参数高效设置下才可行。为此，我们提出了PLUM管道，用于执行数据增强以将对话上采样为问题-答案对，然后利用加权交叉熵损失来微调低秩适应适配器。即使在这个问题的初步探索中，我们的表现也与RAG等基线相当，在100次对话中的准确率达到81.5%。
URL
https://arxiv.org/abs/2411.13405
PDF
-
HyperGAN-CLIP: A Unified Framework for Domain Adaptation, Image Synthesis and Manipulation
2024-11-19 19:36:18Abdul Basit Anees, Ahmet Canberk Baykal, Muhammed Burak Kizil, Duygu Ceylan, Erkut Erdem, Aykut ErdemAbstract
Generative Adversarial Networks (GANs), particularly StyleGAN and its variants, have demonstrated remarkable capabilities in generating highly realistic images. Despite their success, adapting these models to diverse tasks such as domain adaptation, reference-guided synthesis, and text-guided manipulation with limited training data remains challenging. Towards this end, in this study, we present a novel framework that significantly extends the capabilities of a pre-trained StyleGAN by integrating CLIP space via hypernetworks. This integration allows dynamic adaptation of StyleGAN to new domains defined by reference images or textual descriptions. Additionally, we introduce a CLIP-guided discriminator that enhances the alignment between generated images and target domains, ensuring superior image quality. Our approach demonstrates unprecedented flexibility, enabling text-guided image manipulation without the need for text-specific training data and facilitating seamless style transfer. Comprehensive qualitative and quantitative evaluations confirm the robustness and superior performance of our framework compared to existing methods.
Abstract (translated)
生成对抗网络（GANs），特别是StyleGAN及其变体，已经展示了在生成高度逼真图像方面的显著能力。尽管它们取得了成功，但在使用有限的训练数据进行领域适应、参考引导合成和文本引导操作等多样化任务上的应用仍然具有挑战性。为此，在这项研究中，我们提出了一种新颖框架，该框架通过超网络集成CLIP空间来大幅扩展预训练StyleGAN的能力。这种整合使得StyleGAN能够根据参考图像或文本描述动态地适应新领域。此外，我们还引入了由CLIP指导的判别器，以增强生成图像与目标领域的对齐度，确保更优的图像质量。我们的方法展示了前所未有的灵活性，使文本引导的图像操作无需特定于文本的训练数据，并促进了无缝风格迁移。全面的定性和定量评估证实了我们框架相对于现有方法的强大稳健性和卓越性能。
URL
https://arxiv.org/abs/2411.12832
PDF
-
Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning
2024-11-15 12:02:07Yushen Zuo, Jun Xiao, Kin-Chung Chan, Rongkang Dong, Cuixin Yang, Zongqi He, Hao Xie, Kin-Man LamAbstract
The stylization of 3D scenes is an increasingly attractive topic in 3D vision. Although image style transfer has been extensively researched with promising results, directly applying 2D style transfer methods to 3D scenes often fails to preserve the structural and multi-view properties of 3D environments, resulting in unpleasant distortions in images from different viewpoints. To address these issues, we leverage the remarkable generative prior of diffusion-based models and propose a novel style transfer method, OSDiffST, based on a pre-trained one-step diffusion model (i.e., SD-Turbo) for rendering diverse styles in multi-view images of 3D scenes. To efficiently adapt the pre-trained model for multi-view style transfer on small datasets, we introduce a vision condition module to extract style information from the reference style image to serve as conditional input for the diffusion model and employ LoRA in diffusion model for adaptation. Additionally, we consider color distribution alignment and structural similarity between the stylized and content images using two specific loss functions. As a result, our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information. Experiments show that our method surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes. Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. The source code is available at this https URL.
Abstract (translated)
三维场景的风格化在3D视觉领域是一个越来越吸引人的研究主题。尽管图像风格迁移已广泛研究并取得了一定成果，但直接将2D风格迁移方法应用于3D场景往往无法保留3D环境的结构和多视图属性，导致不同视角下的图像出现不愉快的失真现象。为解决这些问题，我们利用基于扩散模型的强大生成先验，并提出一种新的风格迁移方法OSDiffST，该方法基于预训练的一步扩散模型（即SD-Turbo）来渲染3D场景中多视图图像的各种风格。为了高效地将预训练模型适应于小数据集上的多视角风格转移任务，我们引入了视觉条件模块以从参考风格图像中提取样式信息作为扩散模型的条件输入，并采用LoRA在扩散模型中进行适配。此外，我们通过两个特定的损失函数考虑了风格化和内容图像之间的颜色分布对齐与结构相似性。因此，我们的方法能够有效保持风格化图像中的结构信息和多视图一致性，无需任何3D信息。实验表明，我们在为3D场景多视角图像合成各种风格方面超越了其他有前景的风格迁移方法。由我们提出的方法生成的不同视角下的风格化图像具有更优的视觉质量，具备更好的结构性完整性和更低的失真度。源代码可在以下链接获取：[此https URL]。
URL
https://arxiv.org/abs/2411.10130
PDF
-
Artistic Neural Style Transfer Algorithms with Activation Smoothing
2024-11-12 18:44:13Xiangtian Li, Han Cao, Zhaoyang Zhang, Jiacheng Hu, Yuhui Jin, Zihao ZhaoAbstract
The works of Gatys et al. demonstrated the capability of Convolutional Neural Networks (CNNs) in creating artistic style images. This process of transferring content images in different styles is called Neural Style Transfer (NST). In this paper, we re-implement image-based NST, fast NST, and arbitrary NST. We also explore to utilize ResNet with activation smoothing in NST. Extensive experimental results demonstrate that smoothing transformation can greatly improve the quality of stylization results.
Abstract (translated)
Gatys等人的一项研究表明，卷积神经网络（CNN）具备生成艺术风格图像的能力。这种将内容图像转换为不同风格的过程被称为神经样式迁移（NST）。在这篇论文中，我们重新实现了基于图像的NST、快速NST和任意NST。同时，我们也探索了在NST中使用带有激活平滑处理的ResNet的方法。大量的实验结果表明，通过平滑变换可以极大地提升样式的生成质量。
URL
https://arxiv.org/abs/2411.08014
PDF
-
An Interpretable X-ray Style Transfer via Trainable Local Laplacian Filter
2024-11-11 15:47:25Dominik Eckert, Ludwig Ritschl, Christopher Syben, Christian H\"ummer, Julia Wicklein, Marcel Beister, Steffen Kappler, Sebastian StoberAbstract
Radiologists have preferred visual impressions or 'styles' of X-ray images that are manually adjusted to their needs to support their diagnostic performance. In this work, we propose an automatic and interpretable X-ray style transfer by introducing a trainable version of the Local Laplacian Filter (LLF). From the shape of the LLF's optimized remap function, the characteristics of the style transfer can be inferred and reliability of the algorithm can be ensured. Moreover, we enable the LLF to capture complex X-ray style features by replacing the remap function with a Multi-Layer Perceptron (MLP) and adding a trainable normalization layer. We demonstrate the effectiveness of the proposed method by transforming unprocessed mammographic X-ray images into images that match the style of target mammograms and achieve a Structural Similarity Index (SSIM) of 0.94 compared to 0.82 of the baseline LLF style transfer method from Aubry et al.
Abstract (translated)
放射科医生倾向于选择视觉上更符合其需求的手动调整的X光图像风格，以支持他们的诊断表现。在本研究中，我们提出了一种通过引入可训练版本的局部拉普拉斯滤波器（LLF）来实现自动且可解释的X光风格转换方法。从优化后的重映射函数的形状来看，可以推断出风格转换的特点，并确保算法的可靠性。此外，我们将重映射函数替换为多层感知机（MLP），并添加了一个可训练的归一化层，使LLF能够捕捉复杂的X光图像特征。我们通过将未经处理的乳腺X光图像转化为与目标乳腺图像匹配的风格，并达到与Aubry等人提出的基线LLF风格转换方法相比SSIM值为0.94（而基线方法仅为0.82）的结果，证明了所提出方法的有效性。
URL
https://arxiv.org/abs/2411.07072
PDF
-
Diff-MSTC: A Mixing Style Transfer Prototype for Cubase
2024-11-10 19:43:31Soumya Sai Vanka, Lennart Hannink, Jean-Baptiste Rolland, George FazekasAbstract
In our demo, participants are invited to explore the Diff-MSTC prototype, which integrates the Diff-MST model into Steinberg's digital audio workstation (DAW), Cubase. Diff-MST, a deep learning model for mixing style transfer, forecasts mixing console parameters for tracks using a reference song. The system processes up to 20 raw tracks along with a reference song to predict mixing console parameters that can be used to create an initial mix. Users have the option to manually adjust these parameters further for greater control. In contrast to earlier deep learning systems that are limited to research ideas, Diff-MSTC is a first-of-its-kind prototype integrated into a DAW. This integration facilitates mixing decisions on multitracks and lets users input context through a reference song, followed by fine-tuning of audio effects in a traditional manner.
Abstract (translated)
在我们的演示中，参与者被邀请探索将Diff-MST模型集成到Steinberg的数字音频工作站（DAW）Cubase中的Diff-MSTC原型。Diff-MST是一种用于混音风格转换的深度学习模型，它使用参考歌曲来预测音轨的调音台参数。该系统能够处理最多20条原始音轨和一首参考歌曲，以预测可用于创建初步混音的调音台参数。用户可以选择进一步手动调整这些参数，以获得更大的控制权。与仅限于研究概念的早期深度学习系统不同，Diff-MSTC是第一个集成到DAW中的此类原型。这种集成有助于在多轨录音中做出混音决策，并允许用户通过输入参考歌曲来提供上下文信息，随后可以按照传统方式进行音频效果的微调。
URL
https://arxiv.org/abs/2411.06576
PDF
-
AquaFuse: Waterbody Fusion for Physics Guided View Synthesis of Underwater Scenes
2024-11-02 03:20:06Md Abu Bakr Siddique, Jiayi Wu, Ioannis Rekleitis, Md Jahidul IslamAbstract
We introduce the idea of AquaFuse, a physics-based method for synthesizing waterbody properties in underwater imagery. We formulate a closed-form solution for waterbody fusion that facilitates realistic data augmentation and geometrically consistent underwater scene rendering. AquaFuse leverages the physical characteristics of light propagation underwater to synthesize the waterbody from one scene to the object contents of another. Unlike data-driven style transfer, AquaFuse preserves the depth consistency and object geometry in an input scene. We validate this unique feature by comprehensive experiments over diverse underwater scenes. We find that the AquaFused images preserve over 94% depth consistency and 90-95% structural similarity of the input scenes. We also demonstrate that it generates accurate 3D view synthesis by preserving object geometry while adapting to the inherent waterbody fusion process. AquaFuse opens up a new research direction in data augmentation by geometry-preserving style transfer for underwater imaging and robot vision applications.
Abstract (translated)
我们介绍了AquaFuse的概念，这是一种基于物理的方法，用于合成水下图像中的水域属性。我们制定了一个封闭形式的解决方案来融合水域，这有助于实现现实的数据增强和几何上一致的水下场景渲染。AquaFuse利用光在水中传播的物理特性，将一个场景中的水域合成为另一个场景中的物体内容。与数据驱动的风格迁移不同，AquaFuse能够保留输入场景中的深度一致性及对象几何结构。我们通过广泛的实验验证了这一独特特点，在不同的水下场景中进行测试。发现经过AquaFused处理的图像能保持超过94%的深度一致性和90-95%的输入场景结构相似度。我们还展示了它在保留物体几何形状的同时，适应固有的水域融合过程而生成准确的3D视图合成的能力。AquaFuse为数据增强开辟了一个新的研究方向，通过保形风格迁移来支持水下成像和机器人视觉应用。
URL
https://arxiv.org/abs/2411.01119
PDF
-
NCST: Neural-based Color Style Transfer for Video Retouching
2024-11-01 03:25:15Xintao Jiang, Yaosen Chen, Siqin Zhang, Wei Wang, Xuming WenAbstract
Video color style transfer aims to transform the color style of an original video by using a reference style image. Most existing methods employ neural networks, which come with challenges like opaque transfer processes and limited user control over the outcomes. Typically, users cannot fine-tune the resulting images or videos. To tackle this issue, we introduce a method that predicts specific parameters for color style transfer using two images. Initially, we train a neural network to learn the corresponding color adjustment parameters. When applying style transfer to a video, we fine-tune the network with key frames from the video and the chosen style image, generating precise transformation parameters. These are then applied to convert the color style of both images and videos. Our experimental results demonstrate that our algorithm surpasses current methods in color style transfer quality. Moreover, each parameter in our method has a specific, interpretable meaning, enabling users to understand the color style transfer process and allowing them to perform manual fine-tuning if desired.
Abstract (translated)
视频色彩风格迁移旨在通过使用参考风格图像来转换原始视频的色彩风格。现有的大多数方法都采用了神经网络，这带来了诸如转移过程不透明和用户对结果控制有限等挑战。通常情况下，用户无法调整最终生成的图像或视频。为了解决这个问题，我们提出了一种利用两张图片预测特定色彩风格迁移参数的方法。首先，我们训练一个神经网络来学习相应的颜色调整参数。在将风格迁移应用于视频时，我们会使用视频的关键帧和选定的风格图像对网络进行微调，从而生成精确的转换参数。这些参数随后被用于转换图像和视频的颜色风格。实验结果表明，我们的算法在色彩风格迁移的质量上超过了当前的方法。此外，我们方法中的每个参数都有具体的、可解释的意义，使得用户能够理解色彩风格迁移的过程，并允许他们根据需要进行手动微调。
URL
https://arxiv.org/abs/2411.00335
PDF
