Abstract
Diffusion models have become central to various image editing tasks, yet they often fail to fully adhere to physical laws, particularly with effects like shadows, reflections, and occlusions. In this work, we address the challenge of generating photorealistic mirror reflections using diffusion-based generative models. Despite extensive training data, existing diffusion models frequently overlook the nuanced details crucial to authentic mirror reflections. Recent approaches have attempted to resolve this by creating synhetic datasets and framing reflection generation as an inpainting task; however, they struggle to generalize across different object orientations and positions relative to the mirror. Our method overcomes these limitations by introducing key augmentations into the synthetic data pipeline: (1) random object positioning, (2) randomized rotations, and (3) grounding of objects, significantly enhancing generalization across poses and placements. To further address spatial relationships and occlusions in scenes with multiple objects, we implement a strategy to pair objects during dataset generation, resulting in a dataset robust enough to handle these complex scenarios. Achieving generalization to real-world scenes remains a challenge, so we introduce a three-stage training curriculum to develop the MirrorFusion 2.0 model to improve real-world performance. We provide extensive qualitative and quantitative evaluations to support our approach. The project page is available at: this https URL.
Abstract (translated)
扩散模型已经成为各种图像编辑任务的核心，但它们往往未能完全遵循物理定律，尤其是在阴影、反射和遮挡等效果上。在这项工作中，我们解决了使用基于扩散的生成模型生成逼真的镜面反射的问题。尽管训练数据量很大，现有的扩散模型经常会忽略生成真实镜像反射所必需的细微细节。最近的方法试图通过创建合成数据集并将反射生成视为一个像素填充任务来解决这个问题；然而，它们在处理不同对象相对于镜子的位置和朝向时难以泛化。 我们的方法通过引入三个关键增强到合成数据管道中来克服这些限制：（1）随机物体定位，（2）随机旋转以及（3）将物体固定于地面，从而显著增强了跨姿态和位置的泛化能力。为了进一步解决多对象场景中的空间关系和遮挡问题，我们在数据集生成期间实施了一种配对对象的方法，最终形成一个能够处理这些复杂情况的数据集。 实现真实世界场景的泛化仍然是一个挑战，因此我们引入了一个三阶段训练课程来发展MirrorFusion 2.0模型，以提高其在现实环境中的表现。为了支持我们的方法，我们提供了详尽的质量和数量评价结果。项目主页位于：此链接（请根据实际情况替换为实际URL）。
URL
https://arxiv.org/abs/2504.15397
