Abstract
Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.
Abstract (translated)
控制视频与音频生成需要多种模态，包括深度、姿态、摄像机轨迹和音频变换等。然而现有方法要么为固定控制集训练单一整体模型，要么每新增一种模态就需要代价高昂的架构调整。我们提出AVControl，一个基于联合音视频基础模型LTX-2构建的轻量可扩展框架，其中每种控制模态均作为独立LoRA在并行画布上训练，通过注意力层中的额外令牌提供参考信号，除LoRA适配器外无需任何架构改动。我们证明，简单将基于图像的上下文学习方法扩展到视频会导致结构控制失效，而我们的并行画布方法解决了这一问题。在VACE基准测试中，我们在深度与姿态引导生成、图像修复与扩展任务上超越所有评估的基线方法，并在摄像机控制与音视频基准测试中展现竞争力。我们的框架支持多种独立训练的模态：空间对齐控制（如深度、姿态、边缘）、带内参的摄像机轨迹、稀疏运动控制、视频编辑，以及据我们所知首个联合生成模型的模块化音视频控制。我们的方法兼具计算与数据效率：每种模态仅需小型数据集，并在数百至数千次训练步骤内收敛，仅为整体替代方案预算的一小部分。我们公开发布代码与训练好的LoRA检查点。
URL
https://arxiv.org/abs/2603.24793
