Abstract
Aligning the rhythm of visual motion in a video with a given music track is a practical need in multimedia production, yet remains an underexplored task in autonomous video editing. Effective alignment between motion and musical beats enhances viewer engagement and visual appeal, particularly in music videos, promotional content, and cinematic editing. Existing methods typically depend on labor-intensive manual cutting, speed adjustments, or heuristic-based editing techniques to achieve synchronization. While some generative models handle joint video and music generation, they often entangle the two modalities, limiting flexibility in aligning video to music beats while preserving the full visual content. In this paper, we propose a novel and efficient framework, termed MVAA (Music-Video Auto-Alignment), that automatically edits video to align with the rhythm of a given music track while preserving the original visual content. To enhance flexibility, we modularize the task into a two-step process in our MVAA: aligning motion keyframes with audio beats, followed by rhythm-aware video inpainting. Specifically, we first insert keyframes at timestamps aligned with musical beats, then use a frame-conditioned diffusion model to generate coherent intermediate frames, preserving the original video's semantic content. Since comprehensive test-time training can be time-consuming, we adopt a two-stage strategy: pretraining the inpainting module on a small video set to learn general motion priors, followed by rapid inference-time fine-tuning for video-specific adaptation. This hybrid approach enables adaptation within 10 minutes with one epoch on a single NVIDIA 4090 GPU using CogVideoX-5b-I2V as the backbone. Extensive experiments show that our approach can achieve high-quality beat alignment and visual smoothness.
Abstract (translated)
将视频中的视觉运动节奏与给定的音乐曲目对齐是多媒体制作的实际需求，但在自主视频编辑中仍是一个未充分探索的任务。有效地将动作和音乐节拍同步可以增强观众的参与度和视觉吸引力，特别是在音乐视频、宣传内容和电影剪辑方面。现有的方法通常依赖于劳动密集型的手动裁剪、速度调整或基于启发式的编辑技术来实现同步。尽管一些生成模型能够处理联合视频和音乐生成，但它们往往将这两种模式纠缠在一起，限制了在保持完整视觉内容的同时灵活对齐视频与音乐节拍的能力。 为此，在本文中我们提出了一种新颖且高效的框架，称为MVAA（Music-Video Auto-Alignment），该框架可以自动编辑视频以使其节奏与给定的音乐曲目同步，并同时保留原始视觉内容。为了增强灵活性，我们将任务模块化为MVAA中的两步过程：首先将运动关键帧对齐到音频节拍上，然后进行感知节奏的视频修复。具体来说，我们先在时间戳处插入与音乐节拍对齐的关键帧，随后使用受条件约束的扩散模型生成连贯的中间帧，同时保留原始视频的语义内容。 鉴于全面的时间测试训练可能耗时较长，我们采用了一种两阶段策略：首先在一个小规模视频集上预训练修复模块以学习通用运动先验知识，然后进行快速推理时间微调来适应特定视频。这种混合方法能够在单个NVIDIA 4090 GPU上使用CogVideoX-5b-I2V作为骨干网络的情况下，在一个epoch内（10分钟内）完成自适应调整。 广泛的实验表明，我们的方法能够实现高质量的节拍对齐和视觉平滑度。
URL
https://arxiv.org/abs/2506.18881
