Abstract
Music profoundly enhances video production by improving quality, engagement, and emotional resonance, sparking growing interest in video-to-music generation. Despite recent advances, existing approaches remain limited in specific scenarios or undervalue the visual dynamics. To address these limitations, we focus on tackling the complexity of dynamics and resolving temporal misalignment between video and music representations. To this end, we propose DyViM, a novel framework to enhance dynamics modeling for video-to-music generation. Specifically, we extract frame-wise dynamics features via a simplified motion encoder inherited from optical flow methods, followed by a self-attention module for aggregation within frames. These dynamic features are then incorporated to extend existing music tokens for temporal alignment. Additionally, high-level semantics are conveyed through a cross-attention mechanism, and an annealing tuning strategy benefits to fine-tune well-trained music decoders efficiently, therefore facilitating seamless adaptation. Extensive experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.
Abstract (translated)
音乐在视频制作中通过提升质量、增强互动性和情感共鸣，极大地丰富了视频内容，这激发了人们对视频转音乐生成技术的兴趣。尽管最近有了很多进展，现有的方法仍然在特定场景下受到限制或未能充分重视视觉动态的变化。为了解决这些问题，我们专注于处理动态复杂性，并解决视频与音乐表示之间的时序不对齐问题。 为此，我们提出了DyViM（Dynamic Video to Music），这是一个旨在增强视频到音乐生成过程中动态建模的新框架。具体来说，我们通过简化自光学流方法继承而来的运动编码器提取逐帧的动态特征，随后采用自我注意模块进行帧内的聚合操作。这些动态特性被用来扩展现有的音乐令牌以实现时间上的对齐。此外，高层次语义信息则是通过交叉注意力机制传达，并且一个退火调整策略有助于高效地微调预训练好的音乐解码器，从而促进无缝适应。 广泛的实验表明了DyViM相对于现有最先进的（SOTA）方法的优越性。
URL
https://arxiv.org/abs/2504.07594
