Abstract
Text-to-video generation is an emerging field in generative AI, enabling the creation of realistic, semantically accurate videos from text prompts. While current models achieve impressive visual quality and alignment with input text, they typically rely on static knowledge, making it difficult to incorporate new data without retraining from scratch. To address this limitation, we propose VidCLearn, a continual learning framework for diffusion-based text-to-video generation. VidCLearn features a student-teacher architecture where the student model is incrementally updated with new text-video pairs, and the teacher model helps preserve previously learned knowledge through generative replay. Additionally, we introduce a novel temporal consistency loss to enhance motion smoothness and a video retrieval module to provide structural guidance at inference. Our architecture is also designed to be more computationally efficient than existing models while retaining satisfactory generation performance. Experimental results show VidCLearn's superiority over baseline methods in terms of visual quality, semantic alignment, and temporal coherence.
Abstract (translated)
文本到视频生成是生成式人工智能中的一个新兴领域，能够从文字提示中创建出视觉上逼真且语义准确的视频。尽管当前模型在视觉质量和与输入文本的一致性方面取得了令人印象深刻的成绩，但它们通常依赖于静态知识，这使得难以在不重新训练的情况下将新数据纳入系统中。为了解决这一限制，我们提出了VidCLearn，这是一种基于扩散方法的连续学习框架用于文本到视频生成。 VidCLearn采用了一种师生架构，在这种架构下，学生模型会随着新的文字-视频对逐渐更新，而教师模型则通过生成重播来帮助保留之前学到的知识。此外，我们还引入了一种新颖的时间一致性损失函数以增强运动的平滑度，并加入了一个视频检索模块用于在推理时提供结构指导。 我们的框架设计为比现有模型计算效率更高，同时仍能保持满意的生成性能。实验结果表明，与基准方法相比，VidCLearn在视觉质量、语义一致性和时间连贯性方面均表现出优越性。
URL
https://arxiv.org/abs/2509.16956
