Abstract
Conventional 3D style transfer methods rely on a fixed reference image to apply artistic patterns to 3D scenes. However, in practical applications such as virtual or augmented reality, users often prefer more flexible inputs, including textual descriptions and diverse imagery. In this work, we introduce a novel real-time styling technique M2StyleGS to generate a sequence of precisely color-mapped views. It utilizes 3D Gaussian Splatting (3DGS) as a 3D presentation and multi-modality knowledge refined by CLIP as a reference style. M2StyleGS resolves the abnormal transformation issue by employing a precise feature alignment, namely subdivisive flow, it strengthens the projection of the mapped CLIP text-visual combination feature to the VGG style feature. In addition, we introduce observation loss, which assists in the stylized scene better matching the reference style during the generation, and suppression loss, which suppresses the offset of reference color information throughout the decoding process. By integrating these approaches, M2StyleGS can employ text or images as references to generate a set of style-enhanced novel views. Our experiments show that M2StyleGS achieves better visual quality and surpasses the previous work by up to 32.92% in terms of consistency.
Abstract (translated)
传统3D风格迁移方法依赖固定的参考图像将艺术图案应用于3D场景。然而，在虚拟现实或增强现实等实际应用中，用户往往更倾向于使用更灵活的输入方式，包括文本描述和多样化的图像。本研究提出了一种新型实时风格化技术M2StyleGS，用于生成一系列精确着色的视图。该方法以3D高斯泼溅（3DGS）作为三维表征，并利用经CLIP模型提炼的多模态知识作为参考风格。M2StyleGS通过采用精确的特征对齐技术——细分流（subdivisive flow），解决了异常变换问题，同时强化了映射后的CLIP文本-视觉组合特征向VGG风格特征的投影。此外，我们引入了观察损失（observation loss），辅助风格化场景在生成过程中更好地匹配参考风格；并采用抑制损失（suppression loss），在解码全程抑制参考色彩信息的偏移。通过整合这些方法，M2StyleGS能够以文本或图像作为参考，生成一组风格增强的新视角视图。实验表明，M2StyleGS在视觉质量上表现更优，并在一致性指标上较先前工作最高提升32.92%。
URL
https://arxiv.org/abs/2604.03773
