Abstract
Recent advancements in image editing have utilized large-scale multimodal models to enable intuitive, natural instruction-driven interactions. However, conventional methods still face significant challenges, particularly in spatial reasoning, precise region segmentation, and maintaining semantic consistency, especially in complex scenes. To overcome these challenges, we introduce SmartFreeEdit, a novel end-to-end framework that integrates a multimodal large language model (MLLM) with a hypergraph-enhanced inpainting architecture, enabling precise, mask-free image editing guided exclusively by natural language instructions. The key innovations of SmartFreeEdit include:(1)the introduction of region aware tokens and a mask embedding paradigm that enhance the spatial understanding of complex scenes;(2) a reasoning segmentation pipeline designed to optimize the generation of editing masks based on natural language instructions;and (3) a hypergraph-augmented inpainting module that ensures the preservation of both structural integrity and semantic coherence during complex edits, overcoming the limitations of local-based image generation. Extensive experiments on the Reason-Edit benchmark demonstrate that SmartFreeEdit surpasses current state-of-the-art methods across multiple evaluation metrics, including segmentation accuracy, instruction adherence, and visual quality preservation, while addressing the issue of local information focus and improving global consistency in the edited image. Our project will be available at this https URL.
Abstract (translated)
近期的图像编辑技术利用大规模多模态模型实现了直观且自然的语言驱动交互。然而，传统方法在空间推理、精确区域分割以及保持语义一致性方面仍面临重大挑战，尤其是在复杂场景中更为明显。为了解决这些问题，我们引入了SmartFreeEdit——一种结合了超图增强的图像修复架构和多模态大型语言模型（MLLM）的全新端到端框架，它支持仅通过自然语言指令进行无遮罩、精确的图像编辑。SmartFreeEdit的主要创新点包括： 1. 引入区域感知令牌与掩码嵌入范式，以增强对复杂场景的空间理解； 2. 设计了一种推理分割流水线，旨在根据自然语言指令优化生成编辑掩码的过程； 3. 采用超图增强的图像修复模块，在进行复杂编辑时确保结构完整性和语义连贯性的保持，从而克服基于局部信息生成图像的局限性。 在Reason-Edit基准测试中的广泛实验表明，SmartFreeEdit在多个评估指标（包括分割准确性、指令遵循度和视觉质量保留）上均超越了当前最先进的方法，并解决了局部信息关注不足及改进编辑后图像全局一致性的难题。有关我们项目的更多信息可在此URL访问：[https://此链接待补充](https://此链接待补充)。
URL
https://arxiv.org/abs/2504.12704
