arXiv_CV
-
GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning
2025-05-22 17:59:58Chengqi Duan, Rongyao Fang, Yuqing Wang, Kun Wang, Linjiang Huang, Xingyu Zeng, Hongsheng Li, Xihui LiuAbstract
Visual generation models have made remarkable progress in creating realistic images from text prompts, yet struggle with complex prompts that specify multiple objects with precise spatial relationships and attributes. Effective handling of such prompts requires explicit reasoning about the semantic content and spatial layout. We present GoT-R1, a framework that applies reinforcement learning to enhance semantic-spatial reasoning in visual generation. Building upon the Generation Chain-of-Thought approach, GoT-R1 enables models to autonomously discover effective reasoning strategies beyond predefined templates through carefully designed reinforcement learning. To achieve this, we propose a dual-stage multi-dimensional reward framework that leverages MLLMs to evaluate both the reasoning process and final output, enabling effective supervision across the entire generation pipeline. The reward system assesses semantic alignment, spatial accuracy, and visual quality in a unified approach. Experimental results demonstrate significant improvements on T2I-CompBench benchmark, particularly in compositional tasks involving precise spatial relationships and attribute binding. GoT-R1 advances the state-of-the-art in image generation by successfully transferring sophisticated reasoning capabilities to the visual generation domain. To facilitate future research, we make our code and pretrained models publicly available at this https URL.
Abstract (translated)
视觉生成模型在从文本提示创建逼真的图像方面取得了显著进展，但处理包含多个对象及其精确空间关系和属性的复杂提示时仍面临挑战。有效应对这些复杂提示需要对语义内容和空间布局进行明确推理。我们提出了GoT-R1框架，该框架利用强化学习来增强视觉生成中的语义-空间推理能力。基于生成链式思维方法，GoT-R1使模型能够自主发现超越预定义模板的有效推理策略，这得益于精心设计的强化学习机制。 为了实现这一点，我们提出了一种双阶段多维度奖励框架，该框架利用大规模语言模型（MLLMs）来评估推理过程和最终输出，从而在整个生成管道中提供有效的监督。奖赏系统以统一的方式评估语义一致性、空间准确性以及视觉质量。 实验结果显示，在涉及精确空间关系和属性绑定的组合任务上，GoT-R1在T2I-CompBench基准测试中取得了显著改进，尤其是在处理复杂的组成性任务方面。GoT-R1通过成功将高级推理能力转移到图像生成领域，推动了这一领域的最新研究进展。 为了促进未来的相关研究工作，我们已在[此链接](https://此链接提供具体URL)上公开发布了代码和预训练模型。
URL
https://arxiv.org/abs/2505.17022
PDF
-
ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark
2025-05-22 17:59:58Sara Ghaboura, Ketan More, Wafa Alghallabi, Omkar Thawakar, Jorma Laaksonen, Hisham Cholakkal, Salman Khan, Rao Muhammad AnwerAbstract
As Large Multimodal Models (LMMs) become more capable, there is growing interest in evaluating their reasoning processes alongside their final outputs. However, most benchmarks remain focused on English, overlooking languages with rich linguistic and cultural contexts, such as Arabic. To address this gap, we introduce the Comprehensive Arabic Multimodal Reasoning Benchmark (ARB), the first benchmark designed to evaluate step-by-step reasoning in Arabic across both textual and visual modalities. ARB spans 11 diverse domains, including visual reasoning, document understanding, OCR, scientific analysis, and cultural interpretation. It comprises 1,356 multimodal samples paired with 5,119 human-curated reasoning steps and corresponding actions. We evaluated 12 state-of-the-art open- and closed-source LMMs and found persistent challenges in coherence, faithfulness, and cultural grounding. ARB offers a structured framework for diagnosing multimodal reasoning in underrepresented languages and marks a critical step toward inclusive, transparent, and culturally aware AI systems. We release the benchmark, rubric, and evaluation suit to support future research and reproducibility. Code available at: this https URL
Abstract (translated)
随着大型多模态模型（LMMs）能力的提升，人们对评估其推理过程的兴趣也在增加，而不仅仅是关注它们的最终输出。然而，大多数基准测试仍然主要集中在英语上，忽略了具有丰富语言和文化背景的语言，例如阿拉伯语。为了解决这一不足，我们引入了全面的阿拉伯多模态推理基准（ARB），这是第一个旨在通过文本和视觉两种模式评估阿拉伯语分步推理过程的基准。ARB涵盖了包括视觉推理、文档理解、光学字符识别（OCR）、科学分析以及文化解读在内的11个不同领域，并包含了1,356个多模态样本，与之相配的是5,119个人工策划的推理步骤和相应的行为。我们对12种最先进的开源和闭源LMMs进行了评估，发现它们在一致性、忠实度和文化基础方面仍然存在持续性的挑战。 ARB为诊断代表不足的语言中的多模态推理提供了一个结构化的框架，并标志着向包容性更强、更透明且更具文化意识的人工智能系统迈进的关键一步。我们发布了基准测试、评分标准以及用于支持未来研究和可重复性的评估工具。相关代码可在以下网址获取：this https URL
URL
https://arxiv.org/abs/2505.17021
PDF
-
CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms
2025-05-22 17:59:53Shilin Yan, Jiaming Han, Joey Tsai, Hongwei Xue, Rongyao Fang, Lingyi Hong, Ziyu Guo, Ray ZhangAbstract
The advent of Large Multimodal Models (LMMs) has significantly enhanced Large Language Models (LLMs) to process and interpret diverse data modalities (e.g., image and video). However, as input complexity increases, particularly with long video sequences, the number of required tokens has grown significantly, leading to quadratically computational costs. This has made the efficient compression of video tokens in LMMs, while maintaining performance integrity, a pressing research challenge. In this paper, we introduce CrossLMM, decoupling long video sequences from LMMs via a dual cross-attention mechanism, which substantially reduces visual token quantity with minimal performance degradation. Specifically, we first implement a significant token reduction from pretrained visual encoders through a pooling methodology. Then, within LLM layers, we employ a visual-to-visual cross-attention mechanism, wherein the pooled visual tokens function as queries against the original visual token set. This module enables more efficient token utilization while retaining fine-grained informational fidelity. In addition, we introduce a text-to-visual cross-attention mechanism, for which the text tokens are enhanced through interaction with the original visual tokens, enriching the visual comprehension of the text tokens. Comprehensive empirical evaluation demonstrates that our approach achieves comparable or superior performance across diverse video-based LMM benchmarks, despite utilizing substantially fewer computational resources.
Abstract (translated)
大型多模态模型（LMMs）的出现显著增强了大型语言模型（LLMs）处理和解释多样化数据模式（如图像和视频）的能力。然而，随着输入复杂性的增加，尤其是在长视频序列的情况下，所需的token数量大幅增长，导致计算成本呈二次方增长。这使得在保持性能完整性的前提下高效压缩LMMs中的视频token成为一个紧迫的研究挑战。 在本文中，我们介绍了CrossLMM，通过双交叉注意力机制将长视频序列从LMMs中解耦，从而显著减少了视觉token的数量，并且几乎不会对性能产生负面影响。具体来说，我们首先通过对预训练的视觉编码器使用池化方法实现大量token的减少。然后，在LLM层内，我们采用了一种视觉到视觉的交叉注意力机制，其中池化的视觉tokens作为查询与原始视觉token集合进行比较。这一模块使得更高效的token利用成为可能，并且保持了细粒度的信息保真度。 此外，我们引入了一个文本到视觉的交叉注意力机制，在该机制中，文本tokens通过与原始视觉tokens互动而增强，从而丰富了对文本tokens的理解。 全面的经验评估表明，尽管使用了显著较少的计算资源，我们的方法在各种基于视频的LMM基准测试上实现了可比或更优的表现。
URL
https://arxiv.org/abs/2505.17020
PDF
-
Let Androids Dream of Electric Sheep: A Human-like Image Implication Understanding and Reasoning Framework
2025-05-22 17:59:53Chenhao Zhang, Yazhe NiuAbstract
Metaphorical comprehension in images remains a critical challenge for AI systems, as existing models struggle to grasp the nuanced cultural, emotional, and contextual implications embedded in visual content. While multimodal large language models (MLLMs) excel in basic Visual Question Answer (VQA) tasks, they struggle with a fundamental limitation on image implication tasks: contextual gaps that obscure the relationships between different visual elements and their abstract meanings. Inspired by the human cognitive process, we propose Let Androids Dream (LAD), a novel framework for image implication understanding and reasoning. LAD addresses contextual missing through the three-stage framework: (1) Perception: converting visual information into rich and multi-level textual representations, (2) Search: iteratively searching and integrating cross-domain knowledge to resolve ambiguity, and (3) Reasoning: generating context-alignment image implication via explicit reasoning. Our framework with the lightweight GPT-4o-mini model achieves SOTA performance compared to 15+ MLLMs on English image implication benchmark and a huge improvement on Chinese benchmark, performing comparable with the GPT-4o model on Multiple-Choice Question (MCQ) and outperforms 36.7% on Open-Style Question (OSQ). Additionally, our work provides new insights into how AI can more effectively interpret image implications, advancing the field of vision-language reasoning and human-AI interaction. Our project is publicly available at this https URL.
Abstract (translated)
图像中的比喻理解仍然是AI系统的重大挑战，因为现有的模型难以把握视觉内容中嵌入的细腻的文化、情感和上下文含义。尽管多模态大型语言模型（MLLMs）在基本的视觉问答（VQA）任务上表现出色，但在涉及图像内涵的任务方面仍面临一个根本性的限制：即不同视觉元素之间关系及其抽象意义所造成的上下文差距。 受人类认知过程启发，我们提出了一种新的框架——让机器人产生梦境（Let Androids Dream, LAD），旨在理解和推理图像的隐含含义。LAD通过三阶段框架解决上下文缺失的问题：(1)感知：将视觉信息转换为丰富且多层次的文本表示；(2)搜索：迭代地搜索和整合跨域知识以消除歧义；以及(3)推理：通过明确推理生成与背景相符的图像隐含含义。使用轻量级GPT-4o-mini模型，我们的框架在英语图像隐含基准测试中相较于15个以上的MLLMs达到了最先进的性能，并在中国语料库的测试中取得了巨大进步，在多项选择题（MCQ）和开放式风格问题（OSQ）上分别与GPT-4o模型表现相当并超越了后者36.7%。此外，我们的工作为AI如何更有效地解释图像隐含含义提供了新的见解，推动了视觉语言推理及人机交互领域的发展。 本项目已在公开网址上发布：[此链接](https://thishttpsURL.com/)（请将“this https URL”替换为您实际的项目地址）。
URL
https://arxiv.org/abs/2505.17019
PDF
-
SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward
2025-05-22 17:59:53Kaixuan Fan, Kaituo Feng, Haoming Lyu, Dongzhan Zhou, Xiangyu YueAbstract
Recent advances have shown success in eliciting strong reasoning abilities in multimodal large language models (MLLMs) through rule-based reinforcement learning (RL) with outcome rewards. However, this paradigm typically lacks supervision over the thinking process leading to the final this http URL a result, the model may learn sub-optimal reasoning strategies, which can hinder its generalization ability. In light of this, we propose SophiaVL-R1, as an attempt to add reward signals for the thinking process in this paradigm. To achieve this, we first train a thinking reward model that evaluates the quality of the entire thinking process. Given that the thinking reward may be unreliable for certain samples due to reward hacking, we propose the Trust-GRPO method, which assigns a trustworthiness weight to the thinking reward during training. This weight is computed based on the thinking reward comparison of responses leading to correct answers versus incorrect answers, helping to mitigate the impact of potentially unreliable thinking rewards. Moreover, we design an annealing training strategy that gradually reduces the thinking reward over time, allowing the model to rely more on the accurate rule-based outcome reward in later training stages. Experiments show that our SophiaVL-R1 surpasses a series of reasoning MLLMs on various benchmarks (e.g., MathVisita, MMMU), demonstrating strong reasoning and generalization capabilities. Notably, our SophiaVL-R1-7B even outperforms LLaVA-OneVision-72B on most benchmarks, despite the latter having 10 times more parameters. All code, models, and datasets are made publicly available at this https URL.
Abstract (translated)
最近的研究表明，通过基于规则的强化学习（RL）利用结果奖励可以成功地激发多模态大型语言模型（MLLMs）的强大推理能力。然而，这种范式通常缺乏对最终答案形成过程中的思维流程进行监督，因此模型可能会学到次优的推理策略，这会妨碍其泛化能力。为此，我们提出了SophiaVL-R1，试图在这个框架中为思考过程添加奖励信号。 为了实现这一目标，我们首先训练了一个评估整个思维过程质量的思维奖励模型。鉴于某些样本中的思维奖励可能由于奖励操控而不可靠，我们提出了一种Trust-GRPO方法，在此过程中根据导致正确答案和错误答案响应之间的思维奖励比较来分配一个可信度权重。这种方法有助于减轻潜在不准确思维奖励的影响。 此外，我们设计了一个退火训练策略，该策略会随着时间的推移逐渐减少思维奖励的重要性，使得模型在后期的训练阶段能够更多地依赖于准确的结果奖励。实验结果表明，我们的SophiaVL-R1在多个基准测试（如MathVisita、MMMU）上超越了一系列推理MLLMs，展示了强大的推理和泛化能力。特别值得注意的是，在大多数基准测试中，我们的参数量较少的SophiaVL-R1-7B模型甚至超过了具有10倍更多参数的LLaVA-OneVision-72B。 所有代码、模型和数据集均已公开发布在以下链接：[此链接需要您根据原文进行填写]。
URL
https://arxiv.org/abs/2505.17018
PDF
-
Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
2025-05-22 17:59:49Chengzhuo Tong, Ziyu Guo, Renrui Zhang, Wenyu Shan, Xinyu Wei, Zhenghao Xing, Hongsheng Li, Pheng-Ann HengAbstract
Recent advancements underscore the significant role of Reinforcement Learning (RL) in enhancing the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). Two prominent RL algorithms, Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), are central to these developments, showcasing different pros and cons. Autoregressive image generation, also interpretable as a sequential CoT reasoning process, presents unique challenges distinct from LLM-based CoT reasoning. These encompass ensuring text-image consistency, improving image aesthetic quality, and designing sophisticated reward models, rather than relying on simpler rule-based rewards. While recent efforts have extended RL to this domain, these explorations typically lack an in-depth analysis of the domain-specific challenges and the characteristics of different RL strategies. To bridge this gap, we provide the first comprehensive investigation of the GRPO and DPO algorithms in autoregressive image generation, evaluating their in-domain performance and out-of-domain generalization, while scrutinizing the impact of different reward models on their respective capabilities. Our findings reveal that GRPO and DPO exhibit distinct advantages, and crucially, that reward models possessing stronger intrinsic generalization capabilities potentially enhance the generalization potential of the applied RL algorithms. Furthermore, we systematically explore three prevalent scaling strategies to enhance both their in-domain and out-of-domain proficiency, deriving unique insights into efficiently scaling performance for each paradigm. We hope our study paves a new path for inspiring future work on developing more effective RL algorithms to achieve robust CoT reasoning in the realm of autoregressive image generation. Code is released at this https URL
Abstract (translated)
最近的研究进展强调了强化学习（RL）在增强大型语言模型（LLMs）中的链式思维（CoT）推理能力方面的重要作用。两种突出的RL算法，直接偏好优化（DPO）和群体相对策略优化（GRPO），是这些发展中的核心，展示了各自的优点和缺点。自回归图像生成，也可以视为一种序列式的CoT推理过程，提出了不同于基于LLM的CoT推理的独特挑战。这些问题包括确保文本与图像的一致性、提高图像美学质量以及设计复杂的奖励模型，而不是依赖简单的规则基础奖励。虽然最近的努力已经将RL扩展到这一领域，但这些探索通常缺乏对该领域特定挑战和不同RL策略特性的深入分析。 为了填补这一空白，我们首次对自回归图像生成中GRPO和DPO算法进行了全面调查，评估它们在域内性能以及跨域泛化的能力，并审查不同的奖励模型对其能力的影响。我们的研究结果表明，GRPO和DPO展现了各自的独特优势，而且具备更强内在泛化能力的奖励模型可能会提升所应用RL算法的泛化潜力。此外，我们系统地探索了三种流行的扩展策略以增强它们在域内和跨域的能力，并对每个范式的性能扩展得出了独特的见解。 我们希望这项研究为未来开发更有效的RL算法开辟新的道路，在自回归图像生成领域实现稳健的CoT推理。代码可在[此处](https://this_https_URL)获取（请将“this https URL”替换为实际链接）。
URL
https://arxiv.org/abs/2505.17017
PDF
-
Interactive Post-Training for Vision-Language-Action Models
2025-05-22 17:59:45Shuhan Tan, Kairan Dou, Yue Zhao, Philipp Kr\"ahenb\"uhlAbstract
We introduce RIPT-VLA, a simple and scalable reinforcement-learning-based interactive post-training paradigm that fine-tunes pretrained Vision-Language-Action (VLA) models using only sparse binary success rewards. Existing VLA training pipelines rely heavily on offline expert demonstration data and supervised imitation, limiting their ability to adapt to new tasks and environments under low-data regimes. RIPT-VLA addresses this by enabling interactive post-training with a stable policy optimization algorithm based on dynamic rollout sampling and leave-one-out advantage estimation. RIPT-VLA has the following characteristics. First, it applies to various VLA models, resulting in an improvement on the lightweight QueST model by 21.2%, and the 7B OpenVLA-OFT model to an unprecedented 97.5% success rate. Second, it is computationally efficient and data-efficient: with only one demonstration, RIPT-VLA enables an unworkable SFT model (4%) to succeed with a 97% success rate within 15 iterations. Furthermore, we demonstrate that the policy learned by RIPT-VLA generalizes across different tasks and scenarios and is robust to the initial state context. These results highlight RIPT-VLA as a practical and effective paradigm for post-training VLA models through minimal supervision.
Abstract (translated)
我们介绍了一种基于强化学习的简单且可扩展的交互式后期训练范例——RIPT-VLA，该方法仅使用稀疏二元成功奖励对预训练的视觉-语言-动作（VLA）模型进行微调。现有的VLA训练流水线依赖于大量的离线专家演示数据和监督模仿学习，这限制了它们在低数据环境下的适应能力。RIPT-VLA通过启用基于动态采样和留一法优势估计的稳定策略优化算法的交互式后期训练来解决这个问题。 RIPT-VLA具有以下特点：首先，它适用于各种VLA模型，在轻量级QueST模型上提高了21.2%，并且在7B OpenVLA-OFT模型上达到了前所未有的97.5%的成功率。其次，它在计算和数据使用方面都十分高效：仅用一次演示，RIPT-VLA就使原本几乎无法工作的SFT模型（成功率仅为4%）在经过15次迭代后将成功率提高到97%。此外，我们还展示了由RIPT-VLA学习的策略能够跨不同任务和场景进行泛化，并且对初始状态背景具有鲁棒性。 这些结果凸显了RIPT-VLA作为一种通过最小监督有效提升VLA模型后期训练性能的方法的实际价值与效果。
URL
https://arxiv.org/abs/2505.17016
PDF
-
Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models
2025-05-22 17:59:39Runsen Xu, Weiyao Wang, Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Dahua Lin, Matt Feiszli, Kevin J. LiangAbstract
Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images, leaving them ill-suited for robotics and other real-world applications that require multi-frame reasoning. In this paper, we propose a framework to equip MLLMs with robust multi-frame spatial understanding by integrating depth perception, visual correspondence, and dynamic perception. Central to our approach is the MultiSPA dataset, a novel, large-scale collection of more than 27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves significant gains over baselines and proprietary systems, demonstrating scalable, generalizable multi-frame reasoning. We further observe multi-task benefits and early indications of emergent capabilities in challenging scenarios, and showcase how our model can serve as a multi-frame reward annotator for robotics.
Abstract (translated)
多模态大型语言模型（MLLMs）在视觉任务方面取得了迅速进展，但它们的空间理解能力仍局限于单张图像的范畴，这使得这些模型不太适合需要跨多帧进行推理的机器人技术和其他现实世界应用。为此，本文提出了一种框架，旨在通过整合深度感知、视觉对应和动态感知，增强MLLMs在处理多帧数据时的空间理解能力。 我们方法的核心是MultiSPA数据集，这是一个新颖且大规模的数据集合，包含超过2700万个样本，涵盖了各种3D和4D场景。除了MultiSPA数据集之外，我们还提出了一套全面的基准测试框架，在统一的标准下评估一系列空间任务的表现。我们的模型——多帧时空MLLM（Multi-SpatialMLLM），在与基线系统及专有系统的对比中取得了显著的进步，证明了其具备可扩展和通用性的跨多帧推理能力。 此外，我们还观察到了该模型在处理多种任务时的协同效应，并发现了它在面对挑战性场景时所展现出的新颖能力。最后，我们展示了我们的模型如何能够作为机器人技术中的多帧奖励标注器使用。
URL
https://arxiv.org/abs/2505.17015
PDF
-
When Are Concepts Erased From Diffusion Models?
2025-05-22 17:59:09Kevin Lu, Nicky Kriplani, Rohit Gandikota, Minh Pham, David Bau, Chinmay Hegde, Niv CohenAbstract
Concept erasure, the ability to selectively prevent a model from generating specific concepts, has attracted growing interest, with various approaches emerging to address the challenge. However, it remains unclear how thoroughly these methods erase the target concept. We begin by proposing two conceptual models for the erasure mechanism in diffusion models: (i) reducing the likelihood of generating the target concept, and (ii) interfering with the model's internal guidance mechanisms. To thoroughly assess whether a concept has been truly erased from the model, we introduce a suite of independent evaluations. Our evaluation framework includes adversarial attacks, novel probing techniques, and analysis of the model's alternative generations in place of the erased concept. Our results shed light on the tension between minimizing side effects and maintaining robustness to adversarial prompts. Broadly, our work underlines the importance of comprehensive evaluation for erasure in diffusion models.
Abstract (translated)
概念擦除，即模型选择性防止生成特定概念的能力，已引起了越来越多的关注，并且出现了多种方法来应对这一挑战。然而，这些方法是否能够彻底清除目标概念仍不清楚。我们首先提出了两种关于扩散模型中擦除机制的概念模型：(i) 减少生成目标概念的可能性；(ii) 干扰模型的内部引导机制。为了全面评估某个概念是否已被真正从模型中删除，我们引入了一套独立的评估方法。我们的评估框架包括对抗性攻击、新颖的探测技术以及对模型在擦除特定概念后产生的替代内容进行分析。我们的结果揭示了在最小化副作用和保持对抗提示下的鲁棒性之间存在的矛盾。总的来说，我们的工作强调了在扩散模型中进行彻底评估的重要性。
URL
https://arxiv.org/abs/2505.17013
PDF
-
SpatialScore: Towards Unified Evaluation for Multimodal Spatial Understanding
2025-05-22 17:59:03Haoning Wu, Xiao Huang, Yaohui Chen, Ya Zhang, Yanfeng Wang, Weidi XieAbstract
Multimodal large language models (MLLMs) have achieved impressive success in question-answering tasks, yet their capabilities for spatial understanding are less explored. This work investigates a critical question: do existing MLLMs possess 3D spatial perception and understanding abilities? Concretely, we make the following contributions in this paper: (i) we introduce VGBench, a benchmark specifically designed to assess MLLMs for visual geometry perception, e.g., camera pose and motion estimation; (ii) we propose SpatialScore, the most comprehensive and diverse multimodal spatial understanding benchmark to date, integrating VGBench with relevant data from the other 11 existing datasets. This benchmark comprises 28K samples across various spatial understanding tasks, modalities, and QA formats, along with a carefully curated challenging subset, SpatialScore-Hard; (iii) we develop SpatialAgent, a novel multi-agent system incorporating 9 specialized tools for spatial understanding, supporting both Plan-Execute and ReAct reasoning paradigms; (iv) we conduct extensive evaluations to reveal persistent challenges in spatial reasoning while demonstrating the effectiveness of SpatialAgent. We believe SpatialScore will offer valuable insights and serve as a rigorous benchmark for the next evolution of MLLMs.
Abstract (translated)
多模态大型语言模型（MLLMs）在问答任务中取得了令人印象深刻的成功，但它们的空间理解能力却鲜有探索。本研究探讨了一个关键问题：现有的MLLM是否具备三维空间感知和理解的能力？具体而言，本文作出了以下贡献： (i) 我们引入了VGBench，这是一个专门设计的基准测试工具，用于评估MLLM在视觉几何感知方面的能力，例如相机姿态和运动估计； (ii) 我们提出了SpatialScore，这是迄今为止最为全面且多样化的多模态空间理解基准测试，它将VGBench与来自其他11个现有数据集的相关数据进行了整合。该基准测试包含了28,000多个样本，涵盖了各种空间理解任务、模式以及问答格式，并包含了一个精心策划的挑战性子集SpatialScore-Hard； (iii) 我们开发了SpatialAgent，这是一个新型多代理系统，集成有9种专门的空间理解工具，支持计划-执行和反思行动（ReAct）推理范式； (iv) 我们进行了广泛评估，揭示了空间推理中持久存在的挑战，并展示了SpatialAgent的有效性。 我们认为，SpatialScore将提供宝贵的见解，并作为下一代MLLM演进的严格基准。
URL
https://arxiv.org/abs/2505.17012
PDF
-
Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space
2025-05-22 17:59:02Yan Li, Changyao Tian, Renqiu Xia, Ning Liao, Weiwei Guo, Junchi Yan, Hongsheng Li, Jifeng Dai, Hao Li, Xue YangAbstract
We propose AdapTok, an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content. AdapTok is equipped with a block-wise masking strategy that randomly drops tail tokens of each block during training, and a block causal scorer to predict the reconstruction quality of video frames using different numbers of tokens. During inference, an adaptive token allocation strategy based on integer linear programming is further proposed to adjust token usage given predicted scores. Such design allows for sample-wise, content-aware, and temporally dynamic token allocation under a controllable overall budget. Extensive experiments for video reconstruction and generation on UCF-101 and Kinetics-600 demonstrate the effectiveness of our approach. Without additional image data, AdapTok consistently improves reconstruction quality and generation performance under different token budgets, allowing for more scalable and token-efficient generative video modeling.
Abstract (translated)
我们提出了一种自适应时间因果视频分词器AdapTok，它可以根据视频内容灵活地为不同帧分配标记。AdapTok装备有块级别的掩码策略，在训练过程中随机丢弃每个块的尾部标记，并且有一个块因果评分器用于预测使用不同数量令牌时视频帧的重建质量。在推理阶段，我们进一步提出了一种基于整数线性规划的自适应令牌分配策略，以根据预测得分调整令牌使用情况。这种设计允许在可控的整体预算下进行样本级、内容感知和时间动态变化的标记分配。 在UCF-101和Kinetics-600数据集上的大量实验表明了我们方法的有效性。无需额外的图像数据，在不同的令牌预算下，AdapTok能够持续提高重建质量和生成性能，从而允许更可扩展且高效的视频生成建模。
URL
https://arxiv.org/abs/2505.17011
PDF
-
Deep mineralogical segmentation of thin section images based on QEMSCAN maps
2025-05-22 17:58:34Jean Pablo Vieira de Mello, Matheus Augusto Alves Cuglieri, Leandro P. de Figueiredo, Fernando Bordignon, Marcelo Ramalho Albuquerque, Rodrigo Surmas, Bruno Cavalcanti de PaulaAbstract
Interpreting the mineralogical aspects of rock thin sections is an important task for oil and gas reservoirs evaluation. However, human analysis tend to be subjective and laborious. Technologies like QEMSCAN(R) are designed to automate the mineralogical mapping process, but also suffer from limitations like high monetary costs and time-consuming analysis. This work proposes a Convolutional Neural Network model for automatic mineralogical segmentation of thin section images of carbonate rocks. The model is able to mimic the QEMSCAN mapping itself in a low-cost, generalized and efficient manner. For this, the U-Net semantic segmentation architecture is trained on plane and cross polarized thin section images using the corresponding QEMSCAN maps as target, which is an approach not widely explored. The model was instructed to differentiate occurrences of Calcite, Dolomite, Mg-Clay Minerals, Quartz, Pores and the remaining mineral phases as an unique class named "Others", while it was validated on rock facies both seen and unseen during training, in order to address its generalization capability. Since the images and maps are provided in different resolutions, image registration was applied to align then spatially. The study reveals that the quality of the segmentation is very much dependent on these resolution differences and on the variety of learnable rock textures. However, it shows promising results, especially with regard to the proper delineation of minerals boundaries on solid textures and precise estimation of the minerals distributions, describing a nearly linear relationship between expected and predicted distributions, with coefficient of determination (R^2) superior to 0.97 for seen facies and 0.88 for unseen.
Abstract (translated)
岩石薄片的矿物学分析对于油气储层评价是一项重要任务。然而，人工分析往往主观且耗时。虽然诸如QEMSCAN(R)等技术旨在自动化矿物学测绘过程，但它们也存在高成本和耗时的问题。本研究提出了一种基于卷积神经网络（Convolutional Neural Network, CNN）的模型，用于自动分割碳酸盐岩薄片图像中的矿物区域，该方法以低成本、通用且高效的方式模拟了QEMSCAN制图流程。 为了实现这一目标，使用U-Net语义分割架构对平面偏光和交叉偏光下的岩石薄片图像进行训练，并将对应的QEMSCAN地图作为目标。这种基于QEMSCAN地图的训练方式在研究中不常被采用。模型被设定为区分方解石、白云石、镁质粘土矿物、石英、孔隙以及其他矿物相（标记为“Others”）。为了评估其泛化能力，该模型不仅对训练期间见过的岩相进行了验证，还测试了未见过的岩相。 由于图像和地图提供时分辨率不同，应用图像配准技术使它们在空间上对齐。研究表明，分割质量很大程度上取决于这些分辨率差异以及可学习岩石纹理的多样性。然而，研究结果显示出令人鼓舞的结果，特别是在固体纹理中矿物边界划定得当，并且能精确估计矿物分布。该模型预测与实际分布之间呈现出近似线性关系，对于见过和未见岩相分别获得了0.97以上的决定系数（R^2）值。 总的来说，这项工作通过利用深度学习技术提供了低成本、高效率的岩石薄片矿物学分析方法，展示了在油气储层评价中的潜在应用价值。
URL
https://arxiv.org/abs/2505.17008
PDF
-
CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning
2025-05-22 17:58:27Jiange Yang, Yansong Shi, Haoyi Zhu, Mingyu Liu, Kaijing Ma, Yating Wang, Gangshan Wu, Tong He, Limin WangAbstract
Learning latent motion from Internet videos is crucial for building generalist robots. However, existing discrete latent action methods suffer from information loss and struggle with complex and fine-grained dynamics. We propose CoMo, which aims to learn more informative continuous motion representations from diverse, internet-scale videos. CoMo employs a early temporal feature difference mechanism to prevent model collapse and suppress static appearance noise, effectively discouraging shortcut learning problem. Furthermore, guided by the information bottleneck principle, we constrain the latent motion embedding dimensionality to achieve a better balance between retaining sufficient action-relevant information and minimizing the inclusion of action-irrelevant appearance noise. Additionally, we also introduce two new metrics for more robustly and affordably evaluating motion and guiding motion learning methods development: (i) the linear probing MSE of action prediction, and (ii) the cosine similarity between past-to-current and future-to-current motion embeddings. Critically, CoMo exhibits strong zero-shot generalization, enabling it to generate continuous pseudo actions for previously unseen video domains. This capability facilitates unified policy joint learning using pseudo actions derived from various action-less video datasets (such as cross-embodiment videos and, notably, human demonstration videos), potentially augmented with limited labeled robot data. Extensive experiments show that policies co-trained with CoMo pseudo actions achieve superior performance with both diffusion and autoregressive architectures in simulated and real-world settings.
Abstract (translated)
从互联网视频中学习潜在运动对于构建通用型机器人至关重要。然而，现有的离散潜在动作方法存在信息损失的问题，并且难以处理复杂和细微的动态变化。为此我们提出了CoMo（Continuous Motion），旨在从多样化的、大规模的互联网视频中学习更为详尽的连续运动表示。 CoMo采用了早期时间特征差分机制来防止模型崩溃并抑制静态外观噪声，从而有效避免了捷径学习问题的发生。同时，遵循信息瓶颈原则，我们将潜在运动嵌入维度进行限制，以在保留足够的与动作相关的信息和最小化无关的外观噪声之间取得更好的平衡。 此外，我们还引入了两个新的评估指标，用于更加稳健且经济地评价运动并指导运动学习方法的发展：（i）动作预测线性探测MSE；（ii）过去到当前及未来到当前运动嵌入之间的余弦相似度。这两个指标对于衡量模型在不同时间和视角下保持一致性和相关性的能力至关重要。 最关键的是，CoMo展示出了强大的零样本泛化能力，使其能够为之前未见过的视频领域生成连续伪动作。这种能力使得利用从无标签视频数据集中提取的各种伪动作进行统一策略联合学习成为可能（例如跨实体视频和显著的人类演示视频），这在必要时可以结合有限标记的机器人数据进一步增强。 广泛的实验表明，与CoMo伪动作协同训练的策略在模拟和现实世界环境中使用扩散模型和自回归架构均表现出卓越性能。
URL
https://arxiv.org/abs/2505.17006
PDF
-
PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association
2025-05-22 17:57:55Abdul Hannan, Muhammad Arslan Manzoor, Shah Nawaz, Muhammad Irzam Liaqat, Markus Schedl, Mubashir NomanAbstract
We study the task of learning association between faces and voices, which is gaining interest in the multimodal community lately. These methods suffer from the deliberate crafting of negative mining procedures as well as the reliance on the distant margin parameter. These issues are addressed by learning a joint embedding space in which orthogonality constraints are applied to the fused embeddings of faces and voices. However, embedding spaces of faces and voices possess different characteristics and require spaces to be aligned before fusing them. To this end, we propose a method that accurately aligns the embedding spaces and fuses them with an enhanced gated fusion thereby improving the performance of face-voice association. Extensive experiments on the VoxCeleb dataset reveals the merits of the proposed approach.
Abstract (translated)
我们研究了面部与声音之间关联学习的任务，这一任务最近在多模态社区引起了广泛关注。这些方法面临着故意设计负样本挖掘过程以及依赖于远离边际参数的问题。这些问题通过在一个共同嵌入空间中应用正交约束来解决，该空间融合了面部和声音的嵌入表示。然而，面部和声音的嵌入空间具有不同的特性，并且在将它们融合之前需要对齐这些空间。为此，我们提出了一种方法，能够准确地对齐嵌入空间并使用增强型门控融合技术将其融合在一起，从而提高面部与声音关联任务的表现。在VoxCeleb数据集上的广泛实验揭示了所提方法的优势。
URL
https://arxiv.org/abs/2505.17002
PDF
-
Seeing through Satellite Images at Street Views
2025-05-22 17:57:32Ming Qian, Bin Tan, Qiuyu Wang, Xianwei Zheng, Hanjiang Xiong, Gui-Song Xia, Yujun Shen, Nan XueAbstract
This paper studies the task of SatStreet-view synthesis, which aims to render photorealistic street-view panorama images and videos given any satellite image and specified camera positions or trajectories. We formulate to learn neural radiance field from paired images captured from satellite and street viewpoints, which comes to be a challenging learning problem due to the sparse-view natural and the extremely-large viewpoint changes between satellite and street-view images. We tackle the challenges based on a task-specific observation that street-view specific elements, including the sky and illumination effects are only visible in street-view panoramas, and present a novel approach Sat2Density++ to accomplish the goal of photo-realistic street-view panoramas rendering by modeling these street-view specific in neural networks. In the experiments, our method is testified on both urban and suburban scene datasets, demonstrating that Sat2Density++ is capable of rendering photorealistic street-view panoramas that are consistent across multiple views and faithful to the satellite image.
Abstract (translated)
本文研究了SatStreet视图合成任务，该任务旨在根据任何卫星图像和指定的相机位置或轨迹生成逼真的街景全景图像和视频。我们提出了一种从卫星视角和街道视角捕捉到的一对一图像中学习神经辐射场的方法，但由于卫星和街景图像之间的极大幅度视角变化以及稀疏视图自然性的原因，这成为一个具有挑战性的问题。基于特定任务的观察结果，即包括天空和光照效果在内的街景特有元素仅在街景全景中可见，我们提出了一种名为Sat2Density++的新方法，通过在神经网络中建模这些街景特有的元素来实现逼真的街景全景渲染的目标。实验表明，在城市和郊区场景数据集上测试时，我们的方法证明了Sat2Density++能够生成多视角一致且忠实于卫星图像的逼真街景全景图。
URL
https://arxiv.org/abs/2505.17001
PDF
-
Native Segmentation Vision Transformers
2025-05-22 17:55:20Guillem Bras\'o, Aljo\v{s}a O\v{s}ep, Laura Leal-Taix\'eAbstract
Uniform downsampling remains the de facto standard for reducing spatial resolution in vision backbones. In this work, we propose an alternative design built around a content-aware spatial grouping layer, that dynamically assigns tokens to a reduced set based on image boundaries and their semantic content. Stacking our grouping layer across consecutive backbone stages results in hierarchical segmentation that arises natively in the feature extraction process, resulting in our coined Native Segmentation Vision Transformer. We show that a careful design of our architecture enables the emergence of strong segmentation masks solely from grouping layers, that is, without additional segmentation-specific heads. This sets the foundation for a new paradigm of native, backbone-level segmentation, which enables strong zero-shot results without mask supervision, as well as a minimal and efficient standalone model design for downstream segmentation tasks. Our project page is this https URL.
Abstract (translated)
均匀下采样一直是降低视觉骨干网络空间分辨率的事实标准。在本工作中，我们提出了一种基于内容感知的空间分组层的替代设计方案，该设计可以根据图像边界及其语义内容动态地将标记分配给一个缩小的集合中。在整个连续骨干阶段堆叠我们的分组层会产生一种层次化分割，这种分割自然出现在特征提取过程中，从而形成了我们提出的原生分割视觉变换器（Native Segmentation Vision Transformer）。我们展示了对架构进行精心设计可以使仅通过分组层就能产生强大的分割掩码，而无需额外的特定于分割的头部。这为新的原生骨干级分割范式奠定了基础，该范式可以在没有掩码监督的情况下实现强大的零样本结果，并且对于下游分割任务具有最小和高效的独立模型设计。我们的项目页面在此 [URL]。 注：原文中的项目页面链接（https URL）未给出具体网址，在实际引用时需要提供完整的网址信息。
URL
https://arxiv.org/abs/2505.16993
PDF
-
An Effective Training Framework for Light-Weight Automatic Speech Recognition Models
2025-05-22 17:55:09Abdul Hannan, Alessio Brutti, Shah Nawaz, Mubashir NomanAbstract
Recent advancement in deep learning encouraged developing large automatic speech recognition (ASR) models that achieve promising results while ignoring computational and memory constraints. However, deploying such models on low resource devices is impractical despite of their favorable performance. Existing approaches (pruning, distillation, layer skip etc.) transform the large models into smaller ones at the cost of significant performance degradation or require prolonged training of smaller models for better performance. To address these issues, we introduce an efficacious two-step representation learning based approach capable of producing several small sized models from a single large model ensuring considerably better performance in limited number of epochs. Comprehensive experimentation on ASR benchmarks reveals the efficacy of our approach, achieving three-fold training speed-up and up to 12.54% word error rate improvement.
Abstract (translated)
最近的深度学习进展鼓励开发出了一系列大规模自动语音识别（ASR）模型，这些模型在忽略计算和内存限制的情况下取得了令人鼓舞的结果。然而，在资源有限的设备上部署这样的大模型是不切实际的，尽管它们有良好的性能表现。现有的方法（如剪枝、蒸馏、跳过层等），虽然可以将大型模型转换为较小的模型，但会导致显著的性能下降或需要长时间训练小型模型以获得更好的性能。 为了应对这些问题，我们提出了一种有效的两步表示学习方法，可以从单个大规模模型中生成多个小规模模型，并确保在有限的训练周期内有相当不错的性能表现。我们在ASR基准测试上的全面实验表明了该方法的有效性，实现了三倍的训练速度提升，并且错误词率（WER）最多减少了12.54%。
URL
https://arxiv.org/abs/2505.16991
PDF
-
Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
2025-05-22 17:55:04Runpeng Yu, Xinyin Ma, Xinchao WangAbstract
In this work, we propose Dimple, the first Discrete Diffusion Multimodal Large Language Model (DMLLM). We observe that training with a purely discrete diffusion approach leads to significant training instability, suboptimal performance, and severe length bias issues. To address these challenges, we design a novel training paradigm that combines an initial autoregressive phase with a subsequent diffusion phase. This approach yields the Dimple-7B model, trained on the same dataset and using a similar training pipeline as LLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%, demonstrating that DMLLM can achieve performance comparable to that of autoregressive models. To improve inference efficiency, we propose a decoding strategy termed confident decoding, which dynamically adjusts the number of tokens generated at each step, significantly reducing the number of generation iterations. In autoregressive models, the number of forward iterations during generation equals the response length. With confident decoding, however, the number of iterations needed by Dimple is even only $\frac{\text{response length}}{3}$. We also re-implement the prefilling technique in autoregressive models and demonstrate that it does not significantly impact performance on most benchmark evaluations, while offering a speedup of 1.5x to 7x. Additionally, we explore Dimple's capability to precisely control its response using structure priors. These priors enable structured responses in a manner distinct from instruction-based or chain-of-thought prompting, and allow fine-grained control over response format and length, which is difficult to achieve in autoregressive models. Overall, this work validates the feasibility and advantages of DMLLM and enhances its inference efficiency and controllability. Code and models are available at this https URL.
Abstract (translated)
在这项工作中，我们提出了Dimple，这是首个离散扩散多模态大型语言模型（DMLLM）。我们观察到，使用纯粹的离散扩散方法进行训练会导致显著的训练不稳定、次优性能和严重的长度偏差问题。为了应对这些挑战，我们设计了一种新的训练范式，该范式结合了初始自回归阶段与后续的扩散阶段。这种方法生成了Dimple-7B模型，其在相同的语料库上进行了训练，并使用了类似于LLaVA-NEXT的训练管道。最终，Dimple-7B以3.9%的优势超越了LLaVA-NEXT，这表明DMLLM可以实现与自回归模型相当的性能。 为了提高推理效率，我们提出了一种名为“自信解码”的解码策略，该策略在每个步骤中动态调整生成令牌的数量，显著减少了生成迭代次数。在自回归模型中，生成期间的前向迭代次数等于响应长度。然而，在使用自信解码时，Dimple所需的迭代次数仅为响应长度的$\frac{1}{3}$。 此外，我们重新实现了自回归模型中的填充技术，并展示了这种技术对大多数基准评估性能影响不大，但提供了1.5倍到7倍的速度提升。我们也探讨了Dimple利用结构先验精准控制其响应的能力。这些先验使得以不同于指令或链式思考提示的方式生成结构化回复成为可能，从而可以精确地控制回复格式和长度，而这在自回归模型中是难以实现的。 总的来说，这项工作验证了DMLLM的可行性和优势，并提高了它的推理效率和可控性。代码与模型可在[此处](https://this-url.com)获取。
URL
https://arxiv.org/abs/2505.16990
PDF
-
Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation
2025-05-22 17:54:30Moru Liu, Hao Dong, Jessica Kelly, Olga Fink, Mario TrappAbstract
Out-of-distribution (OOD) detection and segmentation are crucial for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. While prior research has primarily focused on unimodal image data, real-world applications are inherently multimodal, requiring the integration of multiple modalities for improved OOD detection. A key challenge is the lack of supervision signals from unknown data, leading to overconfident predictions on OOD samples. To address this challenge, we propose Feature Mixing, an extremely simple and fast method for multimodal outlier synthesis with theoretical support, which can be further optimized to help the model better distinguish between in-distribution (ID) and OOD data. Feature Mixing is modality-agnostic and applicable to various modality combinations. Additionally, we introduce CARLA-OOD, a novel multimodal dataset for OOD segmentation, featuring synthetic OOD objects across diverse scenes and weather conditions. Extensive experiments on SemanticKITTI, nuScenes, CARLA-OOD datasets, and the MultiOOD benchmark demonstrate that Feature Mixing achieves state-of-the-art performance with a $10 \times$ to $370 \times$ speedup. Our source code and dataset will be available at this https URL.
Abstract (translated)
出界（Out-of-distribution，OOD）检测和分割对于在自动驾驶和机器人辅助手术等安全关键应用中部署机器学习模型至关重要。尽管之前的大多数研究主要集中在单模态图像数据上，但现实世界的应用本质上是多模态的，需要整合多种模态以提高OOD检测的效果。一个关键挑战是没有来自未知数据的监督信号，导致模型在处理OOD样本时过于自信。为解决这一挑战，我们提出了特征混合（Feature Mixing）方法，这是一种极其简单且快速的方法，用于生成具有理论支持的多模态异常值，可以通过进一步优化帮助模型更好地区分已知分布（in-distribution，ID）和OOD数据。特征混合与模式无关，并适用于各种模态组合。 此外，我们还介绍了CARLA-OOD，这是一个新颖的多模态数据集，用于OOD分割任务，其中包含在不同场景和天气条件下合成的OOD物体。在SemanticKITTI、nuScenes、CARLA-OOD以及MultiOOD基准测试上进行的大量实验表明，特征混合方法能够实现最先进的性能，并且速度提高了10倍到370倍。我们的源代码和数据集将在[此处](https://this https URL)提供。 该段落翻译为中文后清晰地介绍了研究背景、提出的方法及其优势，以及用于验证新方法的数据集和实验结果。
URL
https://arxiv.org/abs/2505.16985
PDF
-
Pursuing Temporal-Consistent Video Virtual Try-On via Dynamic Pose Interaction
2025-05-22 17:52:34Dong Li, Wenqi Zhong, Wei Yu, Yingwei Pan, Dingwen Zhang, Ting Yao, Junwei Han, Tao MeiAbstract
Video virtual try-on aims to seamlessly dress a subject in a video with a specific garment. The primary challenge involves preserving the visual authenticity of the garment while dynamically adapting to the pose and physique of the subject. While existing methods have predominantly focused on image-based virtual try-on, extending these techniques directly to videos often results in temporal inconsistencies. Most current video virtual try-on approaches alleviate this challenge by incorporating temporal modules, yet still overlook the critical spatiotemporal pose interactions between human and garment. Effective pose interactions in videos should not only consider spatial alignment between human and garment poses in each frame but also account for the temporal dynamics of human poses throughout the entire video. With such motivation, we propose a new framework, namely Dynamic Pose Interaction Diffusion Models (DPIDM), to leverage diffusion models to delve into dynamic pose interactions for video virtual try-on. Technically, DPIDM introduces a skeleton-based pose adapter to integrate synchronized human and garment poses into the denoising network. A hierarchical attention module is then exquisitely designed to model intra-frame human-garment pose interactions and long-term human pose dynamics across frames through pose-aware spatial and temporal attention mechanisms. Moreover, DPIDM capitalizes on a temporal regularized attention loss between consecutive frames to enhance temporal consistency. Extensive experiments conducted on VITON-HD, VVT and ViViD datasets demonstrate the superiority of our DPIDM against the baseline methods. Notably, DPIDM achieves VFID score of 0.506 on VVT dataset, leading to 60.5% improvement over the state-of-the-art GPD-VVTO approach.
Abstract (translated)
视频虚拟试衣的目标是将视频中的主体无缝地穿上特定的衣物。主要挑战在于在动态适应主体姿势和体型的同时，保持服装的真实视觉效果。尽管现有的方法大多集中在基于图像的虚拟试穿技术上，但直接将其应用到视频中通常会导致时间上的不一致性。目前大多数视频虚拟试衣的方法通过加入时间模块来缓解这一问题，但仍忽略了人类与衣物之间的关键时空姿态互动。 为了有效解决视频中的姿势交互，在每一帧中不仅需要考虑人体和衣物姿势的空间对齐，还需要考虑到整个视频中的人体姿势的动态变化。基于此动机，我们提出了一种新的框架——动态姿态互动扩散模型（Dynamic Pose Interaction Diffusion Models, DPIDM），利用扩散模型深入探索动态姿态互动在视频虚拟试衣中的应用。 技术上，DPIDM引入了一个骨架基础的姿态适配器，将同步的人体和衣物姿势整合到去噪网络中。随后设计了一种分层注意力模块，通过基于姿态的空域和时间域注意机制来建模帧内人体与衣物的姿势互动以及跨帧长时间段内的动态变化。此外，DPIDM利用连续帧之间的正则化注意损失来增强时间一致性。 在VITON-HD、VVT 和ViViD 数据集上进行的大量实验表明了我们提出的DPIDM方法相对于基线方法的优势。值得注意的是，在VVT数据集中，DPIDM达到了VFID得分为0.506，比最先进的GPD-VVTO方法提高了60.5%。
URL
https://arxiv.org/abs/2505.16980
PDF
