Abstract
Safety-critical applications, such as autonomous driving and medical image analysis, require extensive multimodal data for rigorous testing. Synthetic data methods are gaining prominence due to the cost and complexity of gathering real-world data, but they demand a high degree of realism and controllability to be useful. This work introduces two novel methods for synthetic data generation in autonomous driving and medical image analysis, namely MObI and AnydoorMed, respectively. MObI is a first-of-its-kind framework for Multimodal Object Inpainting that leverages a diffusion model to produce realistic and controllable object inpaintings across perceptual modalities, demonstrated simultaneously for camera and lidar. Given a single reference RGB image, MObI enables seamless object insertion into existing multimodal scenes at a specified 3D location, guided by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, this approach uses 3D bounding box conditioning to ensure accurate spatial positioning and realistic scaling. AnydoorMed extends this paradigm to the medical imaging domain, focusing on reference-guided inpainting for mammography scans. It leverages a diffusion-based model to inpaint anomalies with impressive detail preservation, maintaining the reference anomaly's structural integrity while semantically blending it with the surrounding tissue. Together, these methods demonstrate that foundation models for reference-guided inpainting in natural images can be readily adapted to diverse perceptual modalities, paving the way for the next generation of systems capable of constructing highly realistic, controllable and multimodal counterfactual scenarios.
Abstract (translated)
安全关键型应用，如自动驾驶和医学影像分析，需要大量的多模态数据来进行严格测试。由于收集现实世界数据的成本高昂且复杂性大，合成数据方法正日益受到重视，但这些方法要求高度的真实感和可控性才能发挥作用。这项工作介绍了两种用于生成自动驾驶和医学影像分析领域合成数据的新方法，分别是MObI（Multimodal Object Inpainting）和AnydoorMed。 MObI是一个开创性的多模态对象修复框架，利用扩散模型跨感知模式产生真实且可控制的对象修复效果，在相机图像和激光雷达数据中同时展示。给定单个参考RGB图像，MObI能够在指定的三维位置将对象无缝插入现有多种模态场景中，并通过边界框指导这一过程，确保语义一致性和多模态连贯性。与仅依赖编辑掩码的传统修复方法不同，这种方法使用3D边界框条件来保证准确的空间定位和逼真的缩放。 AnydoorMed则将这种范式扩展到医学成像领域，专注于基于参考图像的乳腺X光片异常修复。它利用扩散模型在保留参考异常结构完整性的前提下以惊人的细节保持能力对异常进行修复，并使其语义上与周围组织融合一致。 这两种方法共同表明，用于自然图像参考指导修复的基础模型可以轻松适应各种感知模式，为构建高度逼真、可控和多模态的反事实场景铺平了道路。
URL
https://arxiv.org/abs/2507.23058
