Abstract
This paper introduces VimoRAG, a novel video-based retrieval-augmented motion generation framework for motion large language models (LLMs). As motion LLMs face severe out-of-domain/out-of-vocabulary issues due to limited annotated data, VimoRAG leverages large-scale in-the-wild video databases to enhance 3D motion generation by retrieving relevant 2D human motion signals. While video-based motion RAG is nontrivial, we address two key bottlenecks: (1) developing an effective motion-centered video retrieval model that distinguishes human poses and actions, and (2) mitigating the issue of error propagation caused by suboptimal retrieval results. We design the Gemini Motion Video Retriever mechanism and the Motion-centric Dual-alignment DPO Trainer, enabling effective retrieval and generation processes. Experimental results show that VimoRAG significantly boosts the performance of motion LLMs constrained to text-only input.
Abstract (translated)
本文介绍了VimoRAG，这是一种新型的基于视频检索增强的动作生成框架，专门用于运动大型语言模型（LLMs）。由于受限于有限的标注数据，运动LLMs面临着严重的域外/词汇外问题。为此，VimoRAG利用大规模的真实世界视频数据库来通过检索相关的2D人体动作信号提升3D运动生成能力。虽然基于视频的动作检索增强（motion RAG）具有挑战性，我们解决了两个关键瓶颈：(1) 开发一个有效的以动作为中心的视频检索模型，该模型能够区分人类的姿态和行为；以及 (2) 减轻由于次优检索结果导致的误差传播问题。为此，我们设计了双子星运动视频检索器（Gemini Motion Video Retriever）机制和以运动为中心的双重对齐DPO训练器（Motion-centric Dual-alignment DPO Trainer），使有效的检索与生成过程得以实现。实验结果显示，VimoRAG显著提升了仅基于文本输入约束下的运动LLMs的性能。
URL
https://arxiv.org/abs/2508.12081
