Abstract
Customized video generation aims to produce videos that faithfully preserve the subject's appearance from reference images while maintaining temporally consistent motion from reference videos. Existing methods struggle to ensure both subject appearance similarity and motion pattern consistency due to the lack of object-level guidance for subject and motion. To address this, we propose SMRABooth, which leverages the self-supervised encoder and optical flow encoder to provide object-level subject and motion representations. These representations are aligned with the model during the LoRA fine-tuning process. Our approach is structured in three core stages: (1) We exploit subject representations via a self-supervised encoder to guide subject alignment, enabling the model to capture overall structure of subject and enhance high-level semantic consistency. (2) We utilize motion representations from an optical flow encoder to capture structurally coherent and object-level motion trajectories independent of appearance. (3) We propose a subject-motion association decoupling strategy that applies sparse LoRAs injection across both locations and timing, effectively reducing interference between subject and motion LoRAs. Extensive experiments show that SMRABooth excels in subject and motion customization, maintaining consistent subject appearance and motion patterns, proving its effectiveness in controllable text-to-video generation.
Abstract (translated)
定制化的视频生成旨在根据参考图像忠实保留主体外观的同时，依据参考视频保持时间上的一致性动作。现有方法由于缺乏对主体和运动的对象级指导，在确保相似的主体外观与一致的动作模式方面遇到了挑战。为了解决这一问题，我们提出了SMRABooth方案，该方案利用自监督编码器和光学流编码器来提供对象级别的主题和运动表示，并在LoRA微调过程中对这些表示进行校准。我们的方法由三个核心阶段组成： 1. 我们通过一个自监督编码器来使用主体表示，以引导模型捕捉整个主体结构并增强高层次的语义一致性。 2. 采用光学流编码器提供的运动表示，独立于外观特征地捕获结构一致且对象级别的运动轨迹。 3. 提出了一种解耦主题-运动关联策略，在位置和时间上稀疏注入LoRA，有效减少了主体与运动LoRA之间的干扰。 广泛实验表明，SMRABooth在定制化的主题和动作方面表现优异，能够保持一致的主题外观及动作模式，证明了其在可控文本到视频生成中的有效性。
URL
https://arxiv.org/abs/2512.12193
